[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:36 dist-test-slave-zr1t krb5kdc[32672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:36 dist-test-slave-zr1t krb5kdc[32672](info): set up 2 sockets
Aug 12 00:11:36 dist-test-slave-zr1t krb5kdc[32672](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 00:11:38.048635 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:38.164251 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38163
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:38163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:38.245065 32698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:38.245280 32698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:38.245318 32698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:38.246524 32698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:38.246589 32698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:38.246614 32698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:38.246644 32698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:38.246675 32698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:38.248112 32698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:38163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:38.248391 32698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:38.248607 32698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:38.253419 32709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:38.253438 32706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:38.253474 32705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:38.253602 32698 server_base.cc:1034] running on GCE node
I20260812 00:11:38.253904 32698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:38.254168 32698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:38.255304 32698 hybrid_clock.cc:648] HybridClock initialized: now 1786493498255319 us; error 66 us; skew 500 ppm
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:38.257826 32698 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:38.258263 32698 webserver.cc:460] Webserver started at http://127.31.227.62:43543/ using document root <none> and password file <none>
I20260812 00:11:38.258442 32698 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:38.258482 32698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:38.258567 32698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:38.259599 32698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "b47f2cd80fca4f47a8c2201602175c79"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "2d51a5adf18a93206e9f81c379bc4e17"
server_key_iv: "f9061c382d1428695913e3bc352c0050"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.259923 32698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "b47f2cd80fca4f47a8c2201602175c79"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "2d51a5adf18a93206e9f81c379bc4e17"
server_key_iv: "f9061c382d1428695913e3bc352c0050"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.261189 32698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:38.261977 32721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.262173 32698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:11:38.262290 32698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "b47f2cd80fca4f47a8c2201602175c79"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "2d51a5adf18a93206e9f81c379bc4e17"
server_key_iv: "f9061c382d1428695913e3bc352c0050"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.262383 32698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:38.276580 32698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:38.278455 32698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:38.278610 32698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:38.282845 32698 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:38163
I20260812 00:11:38.282893 334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:38163 every 8 connection(s)
I20260812 00:11:38.283229 32698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:38.284024 335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:38.286194 335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79: Bootstrap starting.
I20260812 00:11:38.286806 335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:38.287124 335 log.cc:824] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:38.287828 335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79: No bootstrap required, opened a new log
I20260812 00:11:38.288632 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 32698
I20260812 00:11:38.288712 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:38.289314 32652 external_mini_cluster.cc:1428] Setting key 077b8f87dba0b90a44b5abe95396643d
I20260812 00:11:38.289652 335 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2cd80fca4f47a8c2201602175c79" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38163 } }
I20260812 00:11:38.289799 335 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:38.289876 335 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47f2cd80fca4f47a8c2201602175c79, State: Initialized, Role: FOLLOWER
I20260812 00:11:38.290042 335 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2cd80fca4f47a8c2201602175c79" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38163 } }
I20260812 00:11:38.290159 335 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:38.290222 335 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:38.290320 335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:38.291165 335 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2cd80fca4f47a8c2201602175c79" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38163 } }
I20260812 00:11:38.291282 335 leader_election.cc:302] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47f2cd80fca4f47a8c2201602175c79; no voters:
I20260812 00:11:38.291456 335 leader_election.cc:288] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:38.291499 344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:38.291676 344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [term 1 LEADER]: Becoming Leader. State: Replica: b47f2cd80fca4f47a8c2201602175c79, State: Running, Role: LEADER
I20260812 00:11:38.291697 335 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:38.291810 344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2cd80fca4f47a8c2201602175c79" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38163 } }
I20260812 00:11:38.292241 347 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47f2cd80fca4f47a8c2201602175c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2cd80fca4f47a8c2201602175c79" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38163 } } }
I20260812 00:11:38.292258 350 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47f2cd80fca4f47a8c2201602175c79. Latest consensus state: current_term: 1 leader_uuid: "b47f2cd80fca4f47a8c2201602175c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2cd80fca4f47a8c2201602175c79" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38163 } } }
I20260812 00:11:38.292439 347 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:38.292459 350 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79 [sys.catalog]: This master's current role is: LEADER
W20260812 00:11:38.293763 372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:11:38.293818 372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:11:38.293874 360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:38.294029 360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:11:38.295580 360 catalog_manager.cc:1347] Generated new cluster ID: 4ca8553fb18b4bd088cfa88d99600fd5
I20260812 00:11:38.295650 360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:38.306905 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.290140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.31.227.62:38163)
0812 00:11:38.290312 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:38.290321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:38.290602 (+ 281us) server_negotiation.cc:407] Connection header received
0812 00:11:38.292710 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:38.292725 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:38.292767 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:38.292844 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:38.295136 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.296550 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.297980 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.298601 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.298724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.302840 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:38.302844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:38.302844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:38.302855 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:38.304754 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.305264 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.305267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.305268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.305392 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.305709 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.305710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.305710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.305991 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:38.306124 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:38.306458 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 00:11:38.306639 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260812 00:11:38.309341 360 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:38.310052 360 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:38.328087 360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47f2cd80fca4f47a8c2201602175c79: Generated new TSK 0
I20260812 00:11:38.328261 360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:38.359688 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38163
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:38.442621 378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:38.442821 378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:38.442845 378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:38.444015 378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:38.444068 378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:38.444097 378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:38.445554 378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:38.445755 378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:38.445976 378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:38.451292 389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:38.451324 385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:38.451325 384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:38.451370 378 server_base.cc:1034] running on GCE node
I20260812 00:11:38.451721 378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:38.451983 378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:38.453148 378 hybrid_clock.cc:648] HybridClock initialized: now 1786493498453132 us; error 27 us; skew 500 ppm
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:38.455813 378 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:38.456230 378 webserver.cc:460] Webserver started at http://127.31.227.1:33795/ using document root <none> and password file <none>
I20260812 00:11:38.456481 378 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:38.456558 378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:38.456709 378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:38.457912 378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "231572324de441349a290471860f1df3"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "f1dd70cc5cba054d6745565f20afd957"
server_key_iv: "3a08e68bf61576edeb443460af915d77"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.458294 378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "231572324de441349a290471860f1df3"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "f1dd70cc5cba054d6745565f20afd957"
server_key_iv: "3a08e68bf61576edeb443460af915d77"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.459686 378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:11:38.460466 397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.460664 378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:38.460762 378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "231572324de441349a290471860f1df3"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "f1dd70cc5cba054d6745565f20afd957"
server_key_iv: "3a08e68bf61576edeb443460af915d77"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.460847 378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:38.481148 378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:38.483270 378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:38.483438 378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:38.483686 378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:38.483991 378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:38.484047 378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.484086 378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:38.484133 378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.490278 378 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:40015
I20260812 00:11:38.490332 612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:40015 every 8 connection(s)
I20260812 00:11:38.490635 378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:38.494763 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 378
I20260812 00:11:38.494869 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:38.495054 32652 external_mini_cluster.cc:1428] Setting key dbf75ae676902f674d6f7c750a85f37d
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:38.500626 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.491460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:50579 (local address 127.31.227.62:38163)
0812 00:11:38.491522 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:38.491524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:38.491710 (+ 186us) server_negotiation.cc:407] Connection header received
0812 00:11:38.492347 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:38.492348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:38.492364 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:38.492472 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:38.493417 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.493965 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.494943 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.495209 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.495279 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.497667 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:38.497668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:38.497669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:38.497675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:38.499175 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.499596 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.499598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.499598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.499663 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.499951 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.499952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.499952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.500066 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:38.500139 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:38.500440 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 00:11:38.500537 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:11:38.500783 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.491446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38163 (local address 127.31.227.1:50579)
0812 00:11:38.491595 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:38.491629 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:38.492228 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:38.492547 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:38.492550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:38.493099 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:38.493293 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:38.493297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.494046 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.494046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:38.494818 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:38.494820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.495313 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.495314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:38.495484 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.496070 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:38.496078 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:38.497556 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:38.499270 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:38.499278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:38.499280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:38.499511 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:38.499750 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:38.499751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:38.499751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:38.499868 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:38.500145 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:38.500152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:38.500286 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 00:11:38.500393 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 00:11:38.500554 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260812 00:11:38.501309 614 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38163
I20260812 00:11:38.501420 614 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:38.501645 614 heartbeater.cc:511] Master 127.31.227.62:38163 requested a full tablet report, sending...
I20260812 00:11:38.502242 32747 ts_manager.cc:194] Registered new tserver with Master: 231572324de441349a290471860f1df3 (127.31.227.1:40015)
I20260812 00:11:38.502867 32747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:50579
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:38.521729 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38163
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:38.605733 627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:38.605939 627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:38.605963 627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:38.607123 627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:38.607168 627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:38.607195 627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:38.608636 627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:38.608889 627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:38.609124 627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:38.614135 633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:38.614301 627 server_base.cc:1034] running on GCE node
W20260812 00:11:38.614115 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:38.614115 635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:38.614645 627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:38.614881 627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:38.616045 627 hybrid_clock.cc:648] HybridClock initialized: now 1786493498616031 us; error 28 us; skew 500 ppm
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:38.618666 627 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:38.619086 627 webserver.cc:460] Webserver started at http://127.31.227.2:34995/ using document root <none> and password file <none>
I20260812 00:11:38.619319 627 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:38.619388 627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:38.619542 627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:38.620568 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "42ae95aaa1c34acbaab9ddac85bdd30b"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "20c4b8718e28309c7b4462e7bb889e86"
server_key_iv: "eb28eaae85efeb0750d4c883a3f6b68f"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.620939 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "42ae95aaa1c34acbaab9ddac85bdd30b"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "20c4b8718e28309c7b4462e7bb889e86"
server_key_iv: "eb28eaae85efeb0750d4c883a3f6b68f"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.622429 627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:11:38.623203 653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.623426 627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:38.623539 627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "42ae95aaa1c34acbaab9ddac85bdd30b"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "20c4b8718e28309c7b4462e7bb889e86"
server_key_iv: "eb28eaae85efeb0750d4c883a3f6b68f"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.623615 627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:38.642657 627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:38.644574 627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:38.644731 627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:38.644958 627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:38.645247 627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:38.645300 627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.645368 627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:38.645396 627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.654065 627 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:33227
I20260812 00:11:38.654117 857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:33227 every 8 connection(s)
I20260812 00:11:38.654404 627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:38.656981 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 627
I20260812 00:11:38.657075 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:38.657231 32652 external_mini_cluster.cc:1428] Setting key 0aee925ba4021ab6516e48cd91a2b4ac
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:38.664772 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.655101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:55417 (local address 127.31.227.62:38163)
0812 00:11:38.655171 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:38.655173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:38.655388 (+ 215us) server_negotiation.cc:407] Connection header received
0812 00:11:38.655865 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:38.655866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:38.655878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:38.655932 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:38.656864 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.657661 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.658783 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.659081 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.659151 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.661641 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:38.661642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:38.661643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:38.661650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:38.663116 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.663628 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.663629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.663630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.663694 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.664015 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.664015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.664016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.664120 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:38.664174 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:38.664576 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 00:11:38.664689 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:11:38.665027 860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.655031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38163 (local address 127.31.227.2:55417)
0812 00:11:38.655248 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:38.655282 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:38.655760 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:38.655982 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:38.655985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:38.656562 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:38.656748 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:38.656752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.657752 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.657753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:38.658672 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:38.658675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.659164 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.659165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:38.659264 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.659804 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:38.659814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:38.661483 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:38.663206 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:38.663215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:38.663217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:38.663455 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:38.663774 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:38.663775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:38.663776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:38.663934 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:38.664192 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:38.664220 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:38.664403 (+ 183us) client_negotiation.cc:769] Sending connection context
0812 00:11:38.664553 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 00:11:38.664747 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260812 00:11:38.665508 858 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38163
I20260812 00:11:38.665628 858 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:38.665808 858 heartbeater.cc:511] Master 127.31.227.62:38163 requested a full tablet report, sending...
I20260812 00:11:38.666251 32747 ts_manager.cc:194] Registered new tserver with Master: 42ae95aaa1c34acbaab9ddac85bdd30b (127.31.227.2:33227)
I20260812 00:11:38.666627 32747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:55417
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:38.682502 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38163
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:38.763921 865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:38.764130 865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:38.764173 865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:38.765380 865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:38.765440 865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:38.765489 865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:38.767020 865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:38.767282 865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:38.767522 865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:38.772570 870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:38.772580 878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:38.772661 865 server_base.cc:1034] running on GCE node
W20260812 00:11:38.772579 875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:38.773056 865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:38.773286 865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:38.774441 865 hybrid_clock.cc:648] HybridClock initialized: now 1786493498774448 us; error 44 us; skew 500 ppm
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:38.777428 865 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:38.777889 865 webserver.cc:460] Webserver started at http://127.31.227.3:43069/ using document root <none> and password file <none>
I20260812 00:11:38.778126 865 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:38.778194 865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:38.778334 865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:38.779356 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "9f7eefeb7848444b8385ef42c1091ebd"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "a84cdc3d73f376f66d977c56906f212c"
server_key_iv: "364503fb44d30e113a5a26bb56a71b2f"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.779743 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "9f7eefeb7848444b8385ef42c1091ebd"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "a84cdc3d73f376f66d977c56906f212c"
server_key_iv: "364503fb44d30e113a5a26bb56a71b2f"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.781069 865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:38.781889 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.782071 865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:38.782189 865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "9f7eefeb7848444b8385ef42c1091ebd"
format_stamp: "Formatted at 2026-08-12 00:11:38 on dist-test-slave-zr1t"
server_key: "a84cdc3d73f376f66d977c56906f212c"
server_key_iv: "364503fb44d30e113a5a26bb56a71b2f"
server_key_version: "encryptionkey@0"
I20260812 00:11:38.782274 865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:38.811007 865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:38.812897 865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:38.813060 865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:38.813308 865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:38.813690 865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:38.813753 865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.813803 865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:38.813830 865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:38.820300 865 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41075
I20260812 00:11:38.820358 1095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41075 every 8 connection(s)
I20260812 00:11:38.820658 865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:38.827648 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 865
I20260812 00:11:38.827715 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:38.827914 32652 external_mini_cluster.cc:1428] Setting key 8266f61759d95cdc47bd567cba450b06
I20260812 00:11:38.830986 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.821458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58883 (local address 127.31.227.62:38163)
0812 00:11:38.821568 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:38.821570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:38.821794 (+ 224us) server_negotiation.cc:407] Connection header received
0812 00:11:38.822290 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:38.822291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:38.822301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:38.822366 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:38.823627 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.824133 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.825171 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.825358 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.825416 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.827909 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:38.827910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:38.827911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:38.827919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:38.829576 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.829993 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.829995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.829996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.830054 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.830324 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.830325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.830325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.830448 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:38.830520 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:38.830806 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 00:11:38.830887 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:11:38.831211 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.821432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38163 (local address 127.31.227.3:58883)
0812 00:11:38.821659 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:38.821702 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:38.822191 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:38.822495 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:38.822501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:38.823298 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:38.823508 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:38.823513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.824236 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.824237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:38.825085 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:38.825087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.825466 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.825467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:38.825659 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.826145 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:38.826154 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:38.827786 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:38.829653 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:38.829661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:38.829663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:38.829899 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:38.830131 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:38.830132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:38.830132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:38.830248 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:38.830525 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:38.830533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:38.830672 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 00:11:38.830783 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 00:11:38.830958 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":148,"threads_started":1}
I20260812 00:11:38.831652 1096 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38163
I20260812 00:11:38.831735 1096 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:38.831947 1096 heartbeater.cc:511] Master 127.31.227.62:38163 requested a full tablet report, sending...
I20260812 00:11:38.832326 32747 ts_manager.cc:194] Registered new tserver with Master: 9f7eefeb7848444b8385ef42c1091ebd (127.31.227.3:41075)
I20260812 00:11:38.832729 32747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:58883
I20260812 00:11:38.840008 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:38.864625 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.856818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.31.227.1:40015)
0812 00:11:38.857006 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:38.857018 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:38.857030 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:11:38.857056 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:38.857058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:38.857072 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:38.857146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:38.857732 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.858561 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.859293 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.859632 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.859690 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.861884 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:38.861892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:38.861893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:38.861900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:38.863607 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.863920 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.863923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.863923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.863978 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.864160 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.864161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.864162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.864314 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:38.864451 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:38.864487 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 00:11:38.864551 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
W20260812 00:11:38.864995 449 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34422
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:38.888072 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:38.880925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.31.227.1:40015)
0812 00:11:38.881015 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:38.881018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:38.881101 (+ 83us) server_negotiation.cc:407] Connection header received
0812 00:11:38.881200 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:38.881201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:38.881211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:38.881271 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:38.881941 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.882444 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.883180 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:38.883435 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:38.883498 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:38.885703 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:38.885705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:38.885705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:38.885712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:38.887167 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.887480 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.887481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.887481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.887531 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:38.887693 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:38.887693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:38.887694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:38.887781 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:38.887875 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:38.887944 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 00:11:38.888007 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 00:11:38.888638 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 378
I20260812 00:11:38.905151 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 627
I20260812 00:11:38.912540 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 865
I20260812 00:11:38.920495 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 32698
2026-08-12T00:11:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2199 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[1116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[1116](info): set up 2 sockets
Aug 12 00:11:38 dist-test-slave-zr1t krb5kdc[1116](info): commencing operation
krb5kdc: starting...
W20260812 00:11:40.185783 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:40.302455 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:36503
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:36503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:40.382987 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:40.383174 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:40.383196 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:40.384264 1142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:40.384300 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:40.384315 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:40.384327 1142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:40.384339 1142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:40.385788 1142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:36503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:40.385975 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:40.386135 1142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:40.391222 1153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:40.391194 1156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:40.391433 1142 server_base.cc:1034] running on GCE node
W20260812 00:11:40.391206 1152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:40.391775 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:40.392091 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:40.393249 1142 hybrid_clock.cc:648] HybridClock initialized: now 1786493500393234 us; error 29 us; skew 500 ppm
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:40.395922 1142 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:40.396409 1142 webserver.cc:460] Webserver started at http://127.31.227.62:39771/ using document root <none> and password file <none>
I20260812 00:11:40.396639 1142 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:40.396708 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:40.396853 1142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:40.398007 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "4be0a7d14b064595af48a138c42713a3"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "6c62647edf4024433696396144865441"
server_key_iv: "2e91806902963e32a440626db2450886"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.398375 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "4be0a7d14b064595af48a138c42713a3"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "6c62647edf4024433696396144865441"
server_key_iv: "2e91806902963e32a440626db2450886"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.399696 1142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:40.400484 1167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.400688 1142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:40.400792 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "4be0a7d14b064595af48a138c42713a3"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "6c62647edf4024433696396144865441"
server_key_iv: "2e91806902963e32a440626db2450886"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.400888 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:40.426214 1142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:40.428478 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:40.428650 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:40.432839 1142 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:36503
I20260812 00:11:40.432880 1264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:36503 every 8 connection(s)
I20260812 00:11:40.433262 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:40.434178 1265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:40.436488 1265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3: Bootstrap starting.
I20260812 00:11:40.436954 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1142
I20260812 00:11:40.437052 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:40.437036 1265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:40.437229 32652 external_mini_cluster.cc:1428] Setting key 46484e54f56a0e691cbc134b6eac7e6b
I20260812 00:11:40.437430 1265 log.cc:824] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:40.438279 1265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3: No bootstrap required, opened a new log
I20260812 00:11:40.440707 1265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be0a7d14b064595af48a138c42713a3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 36503 } }
I20260812 00:11:40.440873 1265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:40.440917 1265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be0a7d14b064595af48a138c42713a3, State: Initialized, Role: FOLLOWER
I20260812 00:11:40.441059 1265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be0a7d14b064595af48a138c42713a3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 36503 } }
I20260812 00:11:40.441150 1265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:40.441174 1265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:40.441243 1265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:40.441900 1265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be0a7d14b064595af48a138c42713a3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 36503 } }
I20260812 00:11:40.442009 1265 leader_election.cc:302] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be0a7d14b064595af48a138c42713a3; no voters:
I20260812 00:11:40.442173 1265 leader_election.cc:288] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:40.442237 1273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:40.442431 1265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:40.442461 1273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [term 1 LEADER]: Becoming Leader. State: Replica: 4be0a7d14b064595af48a138c42713a3, State: Running, Role: LEADER
I20260812 00:11:40.442554 1273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be0a7d14b064595af48a138c42713a3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 36503 } }
I20260812 00:11:40.442840 1273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be0a7d14b064595af48a138c42713a3. Latest consensus state: current_term: 1 leader_uuid: "4be0a7d14b064595af48a138c42713a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be0a7d14b064595af48a138c42713a3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 36503 } } }
I20260812 00:11:40.442931 1273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:40.442845 1275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be0a7d14b064595af48a138c42713a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be0a7d14b064595af48a138c42713a3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 36503 } } }
I20260812 00:11:40.443080 1275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:40.443154 1281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:40.443645 1281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:40.446211 1281 catalog_manager.cc:1347] Generated new cluster ID: 8b3ae066d44b42789b8e4373b10531f8
I20260812 00:11:40.446267 1281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:11:40.449113 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.437757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.31.227.62:36503)
0812 00:11:40.437902 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.437915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.437967 (+ 52us) server_negotiation.cc:407] Connection header received
0812 00:11:40.438615 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.438625 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.438684 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.438839 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:40.439416 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.440860 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.441816 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.442461 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.443794 (+ 1333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.445537 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:40.445542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:40.445542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:40.445554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:40.447532 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.447916 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.447919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.447920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.448016 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.448239 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.448240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.448240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.448480 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:40.448645 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.448756 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.448896 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260812 00:11:40.464236 1281 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:40.464923 1281 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:40.470515 1281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be0a7d14b064595af48a138c42713a3: Generated new TSK 0
I20260812 00:11:40.470681 1281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:40.507881 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:36503
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:40.596056 1306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:40.596277 1306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:40.596321 1306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:40.597606 1306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:40.597663 1306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:40.597709 1306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:40.599232 1306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:36503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:40.599515 1306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:40.599746 1306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:40.604847 1315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:40.604877 1314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:40.604847 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:40.605103 1306 server_base.cc:1034] running on GCE node
I20260812 00:11:40.605333 1306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:40.605584 1306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:40.606765 1306 hybrid_clock.cc:648] HybridClock initialized: now 1786493500606750 us; error 34 us; skew 500 ppm
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:40.609328 1306 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:40.609766 1306 webserver.cc:460] Webserver started at http://127.31.227.1:38153/ using document root <none> and password file <none>
I20260812 00:11:40.609958 1306 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:40.610000 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:40.610088 1306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:40.611060 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "da2ed6a8f8214d3ba3790be9daba36b2"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "4e754a6113bdd92a09f80afb5671c719"
server_key_iv: "d732093e4bdaa6648fa39b17029b7dab"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.611348 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "da2ed6a8f8214d3ba3790be9daba36b2"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "4e754a6113bdd92a09f80afb5671c719"
server_key_iv: "d732093e4bdaa6648fa39b17029b7dab"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.612579 1306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:11:40.613287 1330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.613505 1306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:40.613626 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "da2ed6a8f8214d3ba3790be9daba36b2"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "4e754a6113bdd92a09f80afb5671c719"
server_key_iv: "d732093e4bdaa6648fa39b17029b7dab"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.613682 1306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:40.630553 1306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:40.632465 1306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:40.632579 1306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:40.632761 1306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:40.633103 1306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:40.633170 1306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.633213 1306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:40.633244 1306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.639181 1306 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:33831
I20260812 00:11:40.639245 1522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:33831 every 8 connection(s)
I20260812 00:11:40.639599 1306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:40.644018 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1306
I20260812 00:11:40.644094 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:40.644287 32652 external_mini_cluster.cc:1428] Setting key 645f604b3997f30023d220d17c5bed33
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:40.650301 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.640398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:37417 (local address 127.31.227.62:36503)
0812 00:11:40.640485 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.640487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.640623 (+ 136us) server_negotiation.cc:407] Connection header received
0812 00:11:40.641769 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.641770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.641781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.641847 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:40.642510 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.643102 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.644313 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.644629 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.644697 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.647193 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:40.647194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:40.647194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:40.647201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:40.648880 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.649271 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.649273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.649273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.649347 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.649624 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.649626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.649626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.649745 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:40.649842 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.650116 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.650212 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 00:11:40.650498 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.640331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:36503 (local address 127.31.227.1:37417)
0812 00:11:40.640501 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:40.640543 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:40.641669 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:40.641904 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:40.641907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:40.642160 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:40.642409 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.642414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.643187 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.643188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.644227 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.644230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.644734 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.644735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.644890 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.645470 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:40.645479 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:40.647086 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:40.648967 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:40.648974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:40.648976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:40.649187 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:40.649421 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:40.649422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:40.649422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:40.649562 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:40.649834 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:40.649842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:40.649978 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 00:11:40.650082 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 00:11:40.650236 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260812 00:11:40.650933 1523 heartbeater.cc:347] Connected to a master server at 127.31.227.62:36503
I20260812 00:11:40.651023 1523 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:40.651221 1523 heartbeater.cc:511] Master 127.31.227.62:36503 requested a full tablet report, sending...
I20260812 00:11:40.651724 1199 ts_manager.cc:194] Registered new tserver with Master: da2ed6a8f8214d3ba3790be9daba36b2 (127.31.227.1:33831)
I20260812 00:11:40.652344 1199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:37417
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:40.669691 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:36503
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:40.753543 1533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:40.753768 1533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:40.753795 1533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:40.755026 1533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:40.755071 1533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:40.755098 1533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:40.756666 1533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:36503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:40.756942 1533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:40.757191 1533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:40.762564 1547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:40.762574 1551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:40.762610 1533 server_base.cc:1034] running on GCE node
W20260812 00:11:40.762581 1545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:40.763159 1533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:40.763365 1533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:40.764503 1533 hybrid_clock.cc:648] HybridClock initialized: now 1786493500764492 us; error 28 us; skew 500 ppm
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:40.767168 1533 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:40.767545 1533 webserver.cc:460] Webserver started at http://127.31.227.2:38897/ using document root <none> and password file <none>
I20260812 00:11:40.767802 1533 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:40.767880 1533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:40.768038 1533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:40.769079 1533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "c2505dca18044d15ae6521e0c7e11c9c"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "ec6f0fb84e3030319352f1b42360f496"
server_key_iv: "ecde833b3f896ed24d1727347f618def"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.769436 1533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "c2505dca18044d15ae6521e0c7e11c9c"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "ec6f0fb84e3030319352f1b42360f496"
server_key_iv: "ecde833b3f896ed24d1727347f618def"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.770931 1533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:40.771759 1563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.771960 1533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:40.772065 1533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "c2505dca18044d15ae6521e0c7e11c9c"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "ec6f0fb84e3030319352f1b42360f496"
server_key_iv: "ecde833b3f896ed24d1727347f618def"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.772152 1533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:40.790762 1533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:40.793087 1533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:40.793249 1533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:40.793499 1533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:40.793843 1533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:40.793898 1533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.793936 1533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:40.793987 1533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.800161 1533 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:45427
I20260812 00:11:40.800220 1766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:45427 every 8 connection(s)
I20260812 00:11:40.800583 1533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:40.804736 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1533
I20260812 00:11:40.804832 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:40.805018 32652 external_mini_cluster.cc:1428] Setting key c6452592641a1a1bb978db9e094adebc
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:40.811223 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.801374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60049 (local address 127.31.227.62:36503)
0812 00:11:40.801459 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.801461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.801679 (+ 218us) server_negotiation.cc:407] Connection header received
0812 00:11:40.802692 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.802694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.802708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.802761 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:40.803414 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.803929 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.805009 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.805253 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.805321 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.808099 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:40.808101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:40.808101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:40.808108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:40.809685 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.810201 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.810203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.810203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.810272 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.810548 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.810549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.810550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.810673 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:40.810752 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.811048 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.811133 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:11:40.811416 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.801312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:36503 (local address 127.31.227.2:60049)
0812 00:11:40.801511 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:40.801580 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:40.802569 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:40.802829 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:40.802833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:40.803126 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:40.803308 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.803313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.804042 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.804043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.804927 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.804930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.805350 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.805351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.805670 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.806166 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:40.806177 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:40.807982 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:40.809790 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:40.809798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:40.809801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:40.810120 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:40.810350 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:40.810351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:40.810351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:40.810470 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:40.810743 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:40.810751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:40.810902 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 00:11:40.811010 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:11:40.811175 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260812 00:11:40.811969 1767 heartbeater.cc:347] Connected to a master server at 127.31.227.62:36503
I20260812 00:11:40.812053 1767 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:40.812253 1767 heartbeater.cc:511] Master 127.31.227.62:36503 requested a full tablet report, sending...
I20260812 00:11:40.812676 1199 ts_manager.cc:194] Registered new tserver with Master: c2505dca18044d15ae6521e0c7e11c9c (127.31.227.2:45427)
I20260812 00:11:40.813202 1199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:60049
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:40.830590 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:36503
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:40.914614 1779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:40.914883 1779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:40.914933 1779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:40.916478 1779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:40.916548 1779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:40.916586 1779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:40.918201 1779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:36503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:40.918380 1779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:40.918562 1779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:40.923525 1795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:40.923540 1792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:40.923525 1793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:40.923722 1779 server_base.cc:1034] running on GCE node
I20260812 00:11:40.924008 1779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:40.924199 1779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:40.925375 1779 hybrid_clock.cc:648] HybridClock initialized: now 1786493500925361 us; error 25 us; skew 500 ppm
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:40.931043 1779 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:40.931538 1779 webserver.cc:460] Webserver started at http://127.31.227.3:42729/ using document root <none> and password file <none>
I20260812 00:11:40.931766 1779 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:40.931835 1779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:40.931998 1779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:40.933075 1779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "62eef3ed160c4b8c8409af5c6647736a"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "4055b3cc760774180b22eaf49eadc477"
server_key_iv: "39ac66f86d236f23e74165b5a7e7f8db"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.933454 1779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "62eef3ed160c4b8c8409af5c6647736a"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "4055b3cc760774180b22eaf49eadc477"
server_key_iv: "39ac66f86d236f23e74165b5a7e7f8db"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.934865 1779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:40.935676 1807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.935858 1779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:40.935957 1779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "62eef3ed160c4b8c8409af5c6647736a"
format_stamp: "Formatted at 2026-08-12 00:11:40 on dist-test-slave-zr1t"
server_key: "4055b3cc760774180b22eaf49eadc477"
server_key_iv: "39ac66f86d236f23e74165b5a7e7f8db"
server_key_version: "encryptionkey@0"
I20260812 00:11:40.936033 1779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:40.953614 1779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:40.955583 1779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:40.955746 1779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:40.955993 1779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:40.956312 1779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:40.956346 1779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.956368 1779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:40.956418 1779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:40.962898 1779 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:44425
I20260812 00:11:40.962939 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:44425 every 8 connection(s)
I20260812 00:11:40.963291 1779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:11:40.965951 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1779
I20260812 00:11:40.966058 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:40.966208 32652 external_mini_cluster.cc:1428] Setting key 6a7f99e65c2d5e322108c0deb487ee5d
Aug 12 00:11:40 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:40.973169 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.964125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:49867 (local address 127.31.227.62:36503)
0812 00:11:40.964170 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.964172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.964385 (+ 213us) server_negotiation.cc:407] Connection header received
0812 00:11:40.965326 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.965328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.965340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.965404 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:40.966071 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.966772 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.967781 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.968015 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.968093 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.970356 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:40.970358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:40.970358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:40.970371 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:40.971785 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.972203 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.972205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.972205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.972265 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.972531 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.972532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.972533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.972647 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:40.972725 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.972992 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.973078 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:11:40.973363 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.964094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:36503 (local address 127.31.227.3:49867)
0812 00:11:40.964256 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:40.964292 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:40.965211 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:40.965461 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:40.965464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:40.965783 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:40.965970 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.965974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.966859 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.966862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.967677 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.967679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.968102 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.968103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.968223 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.968768 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:40.968777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:40.970203 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:40.971874 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:40.971882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:40.971884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:40.972112 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:40.972340 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:40.972341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:40.972341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:40.972447 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:40.972717 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:40.972725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:40.972862 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 00:11:40.972970 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:11:40.973130 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260812 00:11:40.973933 1997 heartbeater.cc:347] Connected to a master server at 127.31.227.62:36503
I20260812 00:11:40.974032 1997 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:40.974174 1997 heartbeater.cc:511] Master 127.31.227.62:36503 requested a full tablet report, sending...
I20260812 00:11:40.974537 1199 ts_manager.cc:194] Registered new tserver with Master: 62eef3ed160c4b8c8409af5c6647736a (127.31.227.3:44425)
I20260812 00:11:40.974912 1199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:49867
I20260812 00:11:40.977921 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:11:40.984707 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.979424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.31.227.62:36503)
0812 00:11:40.979481 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.979483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.979564 (+ 81us) server_negotiation.cc:407] Connection header received
0812 00:11:40.979601 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.979601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.979610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.979653 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:40.980149 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.980602 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.981265 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.981432 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.981479 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.982655 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:40.982656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:40.982657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:40.982661 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:40.983831 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.984134 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.984135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.984135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.984182 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:40.984310 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:40.984310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:40.984310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:40.984429 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:40.984510 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.984598 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.984652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:11:40.986953 1199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:40.988540 1199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:40.995289 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.990800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:45427 (local address 127.0.0.1:56706)
0812 00:11:40.990922 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:40.990963 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:40.991029 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:40.991485 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:40.991488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:40.991492 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:40.991553 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.991555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.992859 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.992860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.994119 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.994122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.994986 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.994987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.995068 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.995081 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:11:40.995162 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:11:40.995214 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":11,"thread_start_us":75,"threads_started":1}
I20260812 00:11:40.995317 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.991083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.31.227.2:45427)
0812 00:11:40.991283 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.991287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.991315 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:11:40.991352 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.991354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.991367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.991635 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:40.991722 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.992763 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.994211 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.994887 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.994941 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.995006 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.995193 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.995237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260812 00:11:40.995895 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.990808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:44425 (local address 127.0.0.1:60264)
0812 00:11:40.990966 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:40.990974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:40.991029 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:40.991485 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:40.991488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:40.991490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:40.991553 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.991555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.993596 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.993597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.994591 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.994594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.995428 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.995429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.995494 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.995498 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:40.995533 (+ 35us) client_negotiation.cc:240] Negotiation successful
0812 00:11:40.995820 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260812 00:11:40.996263 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.990823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33831 (local address 127.0.0.1:45060)
0812 00:11:40.991495 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:40.991505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:40.991566 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:40.991886 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:40.991886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:40.991888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:40.991942 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.991944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.992803 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.992805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.994119 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:40.994122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.995846 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.995847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:40.995907 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.995913 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:11:40.996156 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 00:11:40.996200 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":72,"thread_start_us":76,"threads_started":1}
I20260812 00:11:40.996276 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.991069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.31.227.3:44425)
0812 00:11:40.991272 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.991277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.991288 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:11:40.991349 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.991351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.991367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.991635 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:40.991708 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.992771 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.994675 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.995320 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.996081 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.996148 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.996174 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.996209 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260812 00:11:40.996268 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51 (DEFAULT_TABLE table=test-table [id=fc67a5ed81e84034b6784be1b0844a9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:40.996568 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:40.991275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.31.227.1:33831)
0812 00:11:40.991473 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:40.991478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:40.991552 (+ 74us) server_negotiation.cc:407] Connection header received
0812 00:11:40.991644 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:40.991647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:40.991668 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:40.991778 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:40.992029 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.992698 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.995039 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:40.995756 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:40.995846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:40.995905 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:40.996441 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 00:11:40.996494 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":127,"threads_started":1}
I20260812 00:11:40.996644 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:40.996866 1885 tablet_service.cc:1467] Processing CreateTablet for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51 (DEFAULT_TABLE table=test-table [id=fc67a5ed81e84034b6784be1b0844a9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:40.997184 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:40.997157 1418 tablet_service.cc:1467] Processing CreateTablet for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51 (DEFAULT_TABLE table=test-table [id=fc67a5ed81e84034b6784be1b0844a9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:40.997464 1418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:40.999864 2022 tablet_bootstrap.cc:492] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: Bootstrap starting.
I20260812 00:11:41.000313 2023 tablet_bootstrap.cc:492] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: Bootstrap starting.
I20260812 00:11:41.000727 2022 tablet_bootstrap.cc:654] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:41.000972 2025 tablet_bootstrap.cc:492] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: Bootstrap starting.
I20260812 00:11:41.001021 2023 tablet_bootstrap.cc:654] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:41.001369 2022 log.cc:824] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:41.001699 2023 log.cc:824] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:41.001830 2025 tablet_bootstrap.cc:654] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:41.002266 2022 tablet_bootstrap.cc:492] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: No bootstrap required, opened a new log
I20260812 00:11:41.002362 2022 ts_tablet_manager.cc:1397] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:11:41.002489 2025 log.cc:824] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:41.003247 2025 tablet_bootstrap.cc:492] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: No bootstrap required, opened a new log
I20260812 00:11:41.003273 2023 tablet_bootstrap.cc:492] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: No bootstrap required, opened a new log
I20260812 00:11:41.003315 2025 ts_tablet_manager.cc:1397] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:11:41.003322 2023 ts_tablet_manager.cc:1397] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:11:41.004103 2022 raft_consensus.cc:348] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.004231 2022 raft_consensus.cc:374] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:41.004276 2022 raft_consensus.cc:729] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62eef3ed160c4b8c8409af5c6647736a, State: Initialized, Role: FOLLOWER
I20260812 00:11:41.004415 2022 consensus_queue.cc:260] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.004694 2022 ts_tablet_manager.cc:1428] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:11:41.004720 1997 heartbeater.cc:503] Master 127.31.227.62:36503 was elected leader, sending a full tablet report...
I20260812 00:11:41.005362 2025 raft_consensus.cc:348] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.005362 2023 raft_consensus.cc:348] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.005549 2023 raft_consensus.cc:374] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:41.005542 2025 raft_consensus.cc:374] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:41.005585 2025 raft_consensus.cc:729] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2ed6a8f8214d3ba3790be9daba36b2, State: Initialized, Role: FOLLOWER
I20260812 00:11:41.005585 2023 raft_consensus.cc:729] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2505dca18044d15ae6521e0c7e11c9c, State: Initialized, Role: FOLLOWER
I20260812 00:11:41.005694 2023 consensus_queue.cc:260] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.005695 2025 consensus_queue.cc:260] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.005965 2025 ts_tablet_manager.cc:1428] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:11:41.005980 1767 heartbeater.cc:503] Master 127.31.227.62:36503 was elected leader, sending a full tablet report...
I20260812 00:11:41.005980 1523 heartbeater.cc:503] Master 127.31.227.62:36503 was elected leader, sending a full tablet report...
I20260812 00:11:41.005965 2023 ts_tablet_manager.cc:1428] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260812 00:11:41.051174 1771 tablet.cc:2369] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:11:41.140370 1525 tablet.cc:2369] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:11:41.156281 2032 raft_consensus.cc:482] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:41.156410 2032 raft_consensus.cc:504] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.156754 2032 leader_election.cc:288] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2505dca18044d15ae6521e0c7e11c9c (127.31.227.2:45427), da2ed6a8f8214d3ba3790be9daba36b2 (127.31.227.1:33831)
I20260812 00:11:41.160526 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.157045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:48457 (local address 127.31.227.1:33831)
0812 00:11:41.157089 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:41.157091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:41.157101 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:41.157141 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:41.157142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:41.157150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:41.157208 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:41.157391 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.158165 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.159580 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.160135 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.160223 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.160231 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:41.160416 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:11:41.160453 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:11:41.160526 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.157032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:44463 (local address 127.31.227.2:45427)
0812 00:11:41.157084 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:41.157086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:41.157275 (+ 189us) server_negotiation.cc:407] Connection header received
0812 00:11:41.157312 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:41.157313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:41.157325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:41.157397 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:41.157605 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.158289 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.159580 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.160161 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.160227 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.160234 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:41.160416 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 00:11:41.160455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:11:41.160532 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.156972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33831 (local address 127.31.227.3:48457)
0812 00:11:41.157020 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:41.157033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:41.157078 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:41.157219 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:41.157221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:41.157223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:41.157304 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:41.157307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.158247 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.158248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:41.159471 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:41.159474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.160242 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.160243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:41.160326 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.160328 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:11:41.160387 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:11:41.160442 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":13}
I20260812 00:11:41.160539 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.156972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:45427 (local address 127.31.227.3:44463)
0812 00:11:41.157198 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:41.157208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:41.157256 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:41.157423 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:41.157424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:41.157425 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:41.157492 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:41.157493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.158364 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.158365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:41.159499 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:41.159501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.160242 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.160243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:41.160323 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.160328 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:11:41.160387 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:11:41.160442 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":60,"thread_start_us":125,"threads_started":1}
I20260812 00:11:41.161074 1683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568fb2a48c6b4fb2b7fe72dd9b428d51" candidate_uuid: "62eef3ed160c4b8c8409af5c6647736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2505dca18044d15ae6521e0c7e11c9c" is_pre_election: true
I20260812 00:11:41.161096 1451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568fb2a48c6b4fb2b7fe72dd9b428d51" candidate_uuid: "62eef3ed160c4b8c8409af5c6647736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" is_pre_election: true
I20260812 00:11:41.161239 1451 raft_consensus.cc:2454] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62eef3ed160c4b8c8409af5c6647736a in term 0.
I20260812 00:11:41.161242 1683 raft_consensus.cc:2454] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62eef3ed160c4b8c8409af5c6647736a in term 0.
I20260812 00:11:41.161500 1811 leader_election.cc:302] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62eef3ed160c4b8c8409af5c6647736a, c2505dca18044d15ae6521e0c7e11c9c; no voters:
I20260812 00:11:41.161695 2032 raft_consensus.cc:2781] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:41.161762 2032 raft_consensus.cc:482] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:41.161780 2032 raft_consensus.cc:3037] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:41.162475 2032 raft_consensus.cc:504] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.162614 2032 leader_election.cc:288] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [CANDIDATE]: Term 1 election: Requested vote from peers c2505dca18044d15ae6521e0c7e11c9c (127.31.227.2:45427), da2ed6a8f8214d3ba3790be9daba36b2 (127.31.227.1:33831)
I20260812 00:11:41.162786 1451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568fb2a48c6b4fb2b7fe72dd9b428d51" candidate_uuid: "62eef3ed160c4b8c8409af5c6647736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da2ed6a8f8214d3ba3790be9daba36b2"
I20260812 00:11:41.162809 1683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568fb2a48c6b4fb2b7fe72dd9b428d51" candidate_uuid: "62eef3ed160c4b8c8409af5c6647736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2505dca18044d15ae6521e0c7e11c9c"
I20260812 00:11:41.162871 1683 raft_consensus.cc:3037] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:41.162890 1451 raft_consensus.cc:3037] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:41.163599 1451 raft_consensus.cc:2454] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P da2ed6a8f8214d3ba3790be9daba36b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62eef3ed160c4b8c8409af5c6647736a in term 1.
I20260812 00:11:41.163638 1683 raft_consensus.cc:2454] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P c2505dca18044d15ae6521e0c7e11c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62eef3ed160c4b8c8409af5c6647736a in term 1.
I20260812 00:11:41.163810 1809 leader_election.cc:302] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62eef3ed160c4b8c8409af5c6647736a, da2ed6a8f8214d3ba3790be9daba36b2; no voters:
I20260812 00:11:41.163908 2032 raft_consensus.cc:2781] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:41.164047 2032 raft_consensus.cc:686] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [term 1 LEADER]: Becoming Leader. State: Replica: 62eef3ed160c4b8c8409af5c6647736a, State: Running, Role: LEADER
I20260812 00:11:41.164165 2032 consensus_queue.cc:237] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } }
I20260812 00:11:41.165081 1199 catalog_manager.cc:5495] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a reported cstate change: term changed from 0 to 1, leader changed from <none> to 62eef3ed160c4b8c8409af5c6647736a (127.31.227.3). New cstate: current_term: 1 leader_uuid: "62eef3ed160c4b8c8409af5c6647736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2505dca18044d15ae6521e0c7e11c9c" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62eef3ed160c4b8c8409af5c6647736a" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 44425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da2ed6a8f8214d3ba3790be9daba36b2" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33831 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:41.179986 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.172253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.31.227.1:33831)
0812 00:11:41.172339 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:41.172342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:41.172448 (+ 106us) server_negotiation.cc:407] Connection header received
0812 00:11:41.172538 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:41.172540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:41.172550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:41.172623 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:41.173217 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.173763 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.174522 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.174713 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.174826 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.177157 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:41.177167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:41.177168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:41.177174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:41.178838 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:41.179178 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:41.179181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:41.179181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:41.179257 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:41.179489 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:41.179490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:41.179490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:41.179628 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:41.179731 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:41.179852 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 00:11:41.179917 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493501, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493501, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:41.204252 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.197263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.31.227.1:33831)
0812 00:11:41.197313 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:41.197316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:41.197404 (+ 88us) server_negotiation.cc:407] Connection header received
0812 00:11:41.197469 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:41.197470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:41.197480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:41.197577 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:41.198107 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.198583 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.199305 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.199500 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.199549 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.201842 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:41.201844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:41.201844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:41.201850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:41.203244 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:41.203587 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:41.203588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:41.203588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:41.203641 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:41.203807 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:41.203808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:41.203808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:41.203895 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:41.203967 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:41.204104 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 00:11:41.204188 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:11:41.213922 1999 tablet.cc:2369] T 568fb2a48c6b4fb2b7fe72dd9b428d51 P 62eef3ed160c4b8c8409af5c6647736a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[1116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:41.228082 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:41.220739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.31.227.1:33831)
0812 00:11:41.220833 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:41.220836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:41.220949 (+ 113us) server_negotiation.cc:407] Connection header received
0812 00:11:41.221039 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:41.221040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:41.221057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:41.221123 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:41.221822 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.222380 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.223126 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:41.223313 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:41.223436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:41.225663 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:41.225665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:41.225666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:41.225673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:41.226869 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:41.227221 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:41.227223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:41.227223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:41.227284 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:41.227536 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:41.227537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:41.227538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:41.227644 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:41.227766 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:41.227898 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 00:11:41.227976 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 00:11:41.228613 1378 tablet_service.cc:2908] Rejecting scan request for tablet 568fb2a48c6b4fb2b7fe72dd9b428d51: Uninitialized: clean time has not yet been initialized
I20260812 00:11:41.229496 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1306
I20260812 00:11:41.247714 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1533
I20260812 00:11:41.263628 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1779
I20260812 00:11:41.274783 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1142
2026-08-12T00:11:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2354 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[2063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[2063](info): set up 2 sockets
Aug 12 00:11:41 dist-test-slave-zr1t krb5kdc[2063](info): commencing operation
krb5kdc: starting...
W20260812 00:11:42.557956 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:42 dist-test-slave-zr1t krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:42.675809 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38649
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:38649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:42.759414 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:42.759630 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:42.759653 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:42.760710 2080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:42.760746 2080 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:11:42.760758 2080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:42.760771 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:42.760784 2080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:42.760797 2080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:42.762259 2080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:38649
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:42.762478 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:42.762650 2080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:42.767491 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:42.767518 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:42.767544 2090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:42.767581 2080 server_base.cc:1034] running on GCE node
I20260812 00:11:42.768002 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:42.768345 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:42.769490 2080 hybrid_clock.cc:648] HybridClock initialized: now 1786493502769473 us; error 44 us; skew 500 ppm
Aug 12 00:11:42 dist-test-slave-zr1t krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493502, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:42.772086 2080 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:42.772548 2080 webserver.cc:460] Webserver started at http://127.31.227.62:45899/ using document root <none> and password file <none>
I20260812 00:11:42.772780 2080 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:42.772853 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:42.773012 2080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:42.774194 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "84e469f1ca314bd48e9a92bf1c5fea3c"
format_stamp: "Formatted at 2026-08-12 00:11:42 on dist-test-slave-zr1t"
server_key: "97246f41aff86e148645aa2cd2438532"
server_key_iv: "602ac43ecbcd5c1824d817b19a803784"
server_key_version: "encryptionkey@0"
I20260812 00:11:42.774577 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "84e469f1ca314bd48e9a92bf1c5fea3c"
format_stamp: "Formatted at 2026-08-12 00:11:42 on dist-test-slave-zr1t"
server_key: "97246f41aff86e148645aa2cd2438532"
server_key_iv: "602ac43ecbcd5c1824d817b19a803784"
server_key_version: "encryptionkey@0"
I20260812 00:11:42.775976 2080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:11:42.776767 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:42.776942 2080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:42.777057 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "84e469f1ca314bd48e9a92bf1c5fea3c"
format_stamp: "Formatted at 2026-08-12 00:11:42 on dist-test-slave-zr1t"
server_key: "97246f41aff86e148645aa2cd2438532"
server_key_iv: "602ac43ecbcd5c1824d817b19a803784"
server_key_version: "encryptionkey@0"
I20260812 00:11:42.777136 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:42.788331 2080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:42.790994 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:42.791178 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:42.796422 2080 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:38649
I20260812 00:11:42.796499 2207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:38649 every 8 connection(s)
I20260812 00:11:42.796833 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:42.797482 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:42.799787 2208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: Bootstrap starting.
I20260812 00:11:42.800405 2208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:42.800709 2208 log.cc:824] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:42.800982 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2080
I20260812 00:11:42.801186 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:42.801394 32652 external_mini_cluster.cc:1428] Setting key bd0e456b85d2443eac6f8006f869af18
I20260812 00:11:42.801468 2208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: No bootstrap required, opened a new log
I20260812 00:11:42.804432 2208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38649 } }
I20260812 00:11:42.804600 2208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:42.804633 2208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e469f1ca314bd48e9a92bf1c5fea3c, State: Initialized, Role: FOLLOWER
I20260812 00:11:42.804747 2208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38649 } }
I20260812 00:11:42.804816 2208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:42.804843 2208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:42.804878 2208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:42.805564 2208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38649 } }
I20260812 00:11:42.805685 2208 leader_election.cc:302] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84e469f1ca314bd48e9a92bf1c5fea3c; no voters:
I20260812 00:11:42.805856 2208 leader_election.cc:288] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:42.805895 2214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:42.806088 2214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [term 1 LEADER]: Becoming Leader. State: Replica: 84e469f1ca314bd48e9a92bf1c5fea3c, State: Running, Role: LEADER
I20260812 00:11:42.806119 2208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:42.806788 2214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38649 } }
I20260812 00:11:42.807502 2216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84e469f1ca314bd48e9a92bf1c5fea3c. Latest consensus state: current_term: 1 leader_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38649 } } }
I20260812 00:11:42.807777 2216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:42.807639 2221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e469f1ca314bd48e9a92bf1c5fea3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38649 } } }
W20260812 00:11:42.808048 2242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:11:42.808092 2242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:11:42.808079 2221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:42.808295 2246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:42.808487 2246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:11:42 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:42.810344 2246 catalog_manager.cc:1347] Generated new cluster ID: 5163c9e33bc448618769b268522bc887
I20260812 00:11:42.810416 2246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:11:42.813764 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:42.801931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.31.227.62:38649)
0812 00:11:42.802135 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:42.802146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:42.802182 (+ 36us) server_negotiation.cc:407] Connection header received
0812 00:11:42.802947 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:42.802956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:42.803038 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:42.803183 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:42.803764 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:42.805142 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:42.806186 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:42.806670 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:42.806775 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:42.810015 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:42.810019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:42.810020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:42.810032 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:42.812008 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:42.812504 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:42.812507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:42.812508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:42.812609 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:42.812832 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:42.812832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:42.812833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:42.813116 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:42.813252 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:42.813404 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 00:11:42.813559 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260812 00:11:42.844681 2246 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:42.845255 2246 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:42.862476 2246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: Generated new TSK 0
I20260812 00:11:42.862702 2246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:42.893846 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38649
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:42.980064 2255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:42.980289 2255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:42.980314 2255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:42.981572 2255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:42.981613 2255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:42.981634 2255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:11:42.981652 2255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:42.983125 2255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:38649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:42.983378 2255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:42.983633 2255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:42.988618 2267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:42.988667 2264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:42.988639 2265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:42.989022 2255 server_base.cc:1034] running on GCE node
I20260812 00:11:42.989259 2255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:42.989495 2255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:42.990694 2255 hybrid_clock.cc:648] HybridClock initialized: now 1786493502990683 us; error 32 us; skew 500 ppm
Aug 12 00:11:42 dist-test-slave-zr1t krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493502, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:42.993269 2255 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:42.993770 2255 webserver.cc:460] Webserver started at http://127.31.227.1:37263/ using document root <none> and password file <none>
I20260812 00:11:42.993953 2255 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:42.993992 2255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:42.994084 2255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:42.995167 2255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "dbdd1df89807483c89f82d22f0ed03dc"
format_stamp: "Formatted at 2026-08-12 00:11:42 on dist-test-slave-zr1t"
server_key: "7c15b86f09e0c4af3d248cfba124b86a"
server_key_iv: "ffaed12b132cb9aaa15ada142eac849b"
server_key_version: "encryptionkey@0"
I20260812 00:11:42.995471 2255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "dbdd1df89807483c89f82d22f0ed03dc"
format_stamp: "Formatted at 2026-08-12 00:11:42 on dist-test-slave-zr1t"
server_key: "7c15b86f09e0c4af3d248cfba124b86a"
server_key_iv: "ffaed12b132cb9aaa15ada142eac849b"
server_key_version: "encryptionkey@0"
I20260812 00:11:42.996711 2255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:42.997355 2286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:42.997548 2255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:42.997635 2255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "dbdd1df89807483c89f82d22f0ed03dc"
format_stamp: "Formatted at 2026-08-12 00:11:42 on dist-test-slave-zr1t"
server_key: "7c15b86f09e0c4af3d248cfba124b86a"
server_key_iv: "ffaed12b132cb9aaa15ada142eac849b"
server_key_version: "encryptionkey@0"
I20260812 00:11:42.997689 2255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:43.020061 2255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:43.022198 2255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:43.022396 2255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:43.022970 2255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:43.023034 2255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.023072 2255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:43.023100 2255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.030794 2255 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:35781
I20260812 00:11:43.030916 2480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:35781 every 8 connection(s)
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493502, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:43.031224 2255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:43.034221 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.023681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:42281 (local address 127.31.227.62:38649)
0812 00:11:43.023749 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.023751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.023978 (+ 227us) server_negotiation.cc:407] Connection header received
0812 00:11:43.024613 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.024615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.024628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.024713 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.025751 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.026619 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.028329 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.028657 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.028715 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.031689 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.031691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.031692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.031700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.032870 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.033256 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.033257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.033258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.033319 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.033595 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.033596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.033596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.033743 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.033801 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.034071 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.034153 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:11:43.034344 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.023677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.31.227.1:42281)
0812 00:11:43.023851 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.023889 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.024498 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.024860 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.024866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.025284 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:43.025627 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.025635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.026742 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.026743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.028221 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.028225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.028745 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.028747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.028920 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.029505 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:43.029544 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:43.031595 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:43.032944 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.032951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.032953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.033168 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.033394 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.033394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.033395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.033492 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.033806 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:43.033812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:43.033948 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.034028 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.034147 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260812 00:11:43.034942 2482 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38649
I20260812 00:11:43.035061 2482 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:43.035243 2482 heartbeater.cc:511] Master 127.31.227.62:38649 requested a full tablet report, sending...
I20260812 00:11:43.035867 2128 ts_manager.cc:194] Registered new tserver with Master: dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781)
I20260812 00:11:43.036790 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:42281
I20260812 00:11:43.039372 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2255
I20260812 00:11:43.039490 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:43.039687 32652 external_mini_cluster.cc:1428] Setting key 563f924523caee85170ea6d18b0e9240
I20260812 00:11:43.041829 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.036054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.31.227.1:34123)
0812 00:11:43.036211 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.036222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.036284 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.036404 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.036405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.036614 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:43.036673 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.036674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.037449 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.037450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.038069 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.038072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.038371 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.038372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.038422 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.038753 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:43.038757 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:43.039019 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:43.040773 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.040774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.040775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.040950 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.041167 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.041167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.041167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.041219 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.041488 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:43.041488 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:43.041571 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.041641 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.041746 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260812 00:11:43.041901 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.036137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:34123 (local address 127.31.227.62:38649)
0812 00:11:43.036182 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.036184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.036276 (+ 92us) server_negotiation.cc:407] Connection header received
0812 00:11:43.036339 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.036339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.036347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.036399 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.036745 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.037348 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.038140 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.038316 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.038371 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.039092 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.039094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.039094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.039100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.040692 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.041022 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.041023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.041023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.041101 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.041280 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.041281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.041282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.041424 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.041490 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.041690 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.041775 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:43.068729 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38649
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:43.152469 2500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:43.152712 2500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:43.152751 2500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:43.153937 2500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:43.154008 2500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:43.154042 2500 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:11:43.154080 2500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:43.155512 2500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:38649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:43.155784 2500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:43.156028 2500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:43.161053 2518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:43.161080 2513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:43.161060 2522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:43.161339 2500 server_base.cc:1034] running on GCE node
I20260812 00:11:43.161602 2500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:43.161901 2500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:43.163065 2500 hybrid_clock.cc:648] HybridClock initialized: now 1786493503163048 us; error 47 us; skew 500 ppm
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:43.165704 2500 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:43.166128 2500 webserver.cc:460] Webserver started at http://127.31.227.2:35507/ using document root <none> and password file <none>
I20260812 00:11:43.166388 2500 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:43.166460 2500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:43.166620 2500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:43.167685 2500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "a74e637f47b947beacdbdeded61349b8"
format_stamp: "Formatted at 2026-08-12 00:11:43 on dist-test-slave-zr1t"
server_key: "5add4940125b7e431789410ee160f96b"
server_key_iv: "0053853a7dba52d5005eed77385a8296"
server_key_version: "encryptionkey@0"
I20260812 00:11:43.168067 2500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "a74e637f47b947beacdbdeded61349b8"
format_stamp: "Formatted at 2026-08-12 00:11:43 on dist-test-slave-zr1t"
server_key: "5add4940125b7e431789410ee160f96b"
server_key_iv: "0053853a7dba52d5005eed77385a8296"
server_key_version: "encryptionkey@0"
I20260812 00:11:43.169478 2500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:43.170380 2535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.170576 2500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:43.170673 2500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "a74e637f47b947beacdbdeded61349b8"
format_stamp: "Formatted at 2026-08-12 00:11:43 on dist-test-slave-zr1t"
server_key: "5add4940125b7e431789410ee160f96b"
server_key_iv: "0053853a7dba52d5005eed77385a8296"
server_key_version: "encryptionkey@0"
I20260812 00:11:43.170758 2500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:43.177654 2500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:43.179723 2500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:43.179874 2500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:43.180369 2500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:43.180434 2500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.180470 2500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:43.180500 2500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:43.188273 2500 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:45177
I20260812 00:11:43.188650 2500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:43.188699 2742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:45177 every 8 connection(s)
I20260812 00:11:43.190652 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.180902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:50085 (local address 127.31.227.62:38649)
0812 00:11:43.180974 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.180977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.181231 (+ 254us) server_negotiation.cc:407] Connection header received
0812 00:11:43.181814 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.181815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.181830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.181916 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.182579 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.183444 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.184503 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.184846 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.184936 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.187794 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.187796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.187796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.187803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.189396 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.189842 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.189843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.189843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.189898 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.190124 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.190125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.190125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.190232 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.190281 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.190525 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.190589 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:11:43.190809 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.180920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.31.227.2:50085)
0812 00:11:43.181100 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.181140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.181715 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.181998 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.182001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.182308 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:43.182499 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.182504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.183545 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.183546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.184392 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.184395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.184957 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.184959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.185101 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.185673 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:43.185685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:43.187680 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:43.189471 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.189476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.189477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.189766 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.189963 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.189963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.189963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.190061 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.190286 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:43.190290 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:43.190408 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.190484 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.190602 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260812 00:11:43.191383 2743 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38649
I20260812 00:11:43.191480 2743 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:43.191651 2743 heartbeater.cc:511] Master 127.31.227.62:38649 requested a full tablet report, sending...
I20260812 00:11:43.192067 2128 ts_manager.cc:194] Registered new tserver with Master: a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177)
I20260812 00:11:43.192540 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:50085
I20260812 00:11:43.193804 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2500
I20260812 00:11:43.193899 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:43.194108 32652 external_mini_cluster.cc:1428] Setting key 70f7636a387154693da36b24cb4ad341
I20260812 00:11:43.198622 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.192435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.31.227.2:46549)
0812 00:11:43.192561 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.192572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.192632 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.192792 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.192793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.192973 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:43.193043 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.193045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.193812 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.193812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.194577 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.194580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.195041 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.195042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.195138 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.195601 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:43.195605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:43.195874 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:43.197487 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.197488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.197488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.197750 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.197962 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.197963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.197964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.198031 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.198387 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:43.198387 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:43.198425 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.198478 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.198553 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260812 00:11:43.198719 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.192514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46549 (local address 127.31.227.62:38649)
0812 00:11:43.192557 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.192559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.192616 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:11:43.192702 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.192703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.192715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.192779 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.193137 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.193713 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.194646 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.194936 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.195011 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.195969 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.195971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.195971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.195980 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.197396 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.197844 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.197845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.197845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.197897 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.198097 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.198098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.198099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.198293 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.198399 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.198506 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.198588 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:43.220450 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38649
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:43.305616 2757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:43.305881 2757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:43.305908 2757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:43.307199 2757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:43.307238 2757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:43.307281 2757 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:11:43.307323 2757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:43.308943 2757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:38649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:43.309119 2757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:43.309289 2757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:43.314325 2768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:43.314311 2764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:43.314442 2757 server_base.cc:1034] running on GCE node
W20260812 00:11:43.314306 2763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:43.314769 2757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:43.315007 2757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:43.316148 2757 hybrid_clock.cc:648] HybridClock initialized: now 1786493503316127 us; error 38 us; skew 500 ppm
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:43.318640 2757 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:43.319095 2757 webserver.cc:460] Webserver started at http://127.31.227.3:42785/ using document root <none> and password file <none>
I20260812 00:11:43.319272 2757 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:43.319311 2757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:43.319418 2757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:43.320393 2757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "492c8da422c04d3d8590796b752bd4bc"
format_stamp: "Formatted at 2026-08-12 00:11:43 on dist-test-slave-zr1t"
server_key: "6f8b57a98196479454b2fae01ac67afb"
server_key_iv: "1ccf1333b453f09d883110d5fa045171"
server_key_version: "encryptionkey@0"
I20260812 00:11:43.320693 2757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "492c8da422c04d3d8590796b752bd4bc"
format_stamp: "Formatted at 2026-08-12 00:11:43 on dist-test-slave-zr1t"
server_key: "6f8b57a98196479454b2fae01ac67afb"
server_key_iv: "1ccf1333b453f09d883110d5fa045171"
server_key_version: "encryptionkey@0"
I20260812 00:11:43.322127 2757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:11:43.322832 2782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.323025 2757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:43.323118 2757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "492c8da422c04d3d8590796b752bd4bc"
format_stamp: "Formatted at 2026-08-12 00:11:43 on dist-test-slave-zr1t"
server_key: "6f8b57a98196479454b2fae01ac67afb"
server_key_iv: "1ccf1333b453f09d883110d5fa045171"
server_key_version: "encryptionkey@0"
I20260812 00:11:43.323195 2757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:43.333317 2757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:43.335183 2757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:43.335348 2757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:43.335909 2757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:43.335949 2757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.335970 2757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:43.335983 2757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:43.344061 2757 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:39497
I20260812 00:11:43.344134 2967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:39497 every 8 connection(s)
I20260812 00:11:43.344483 2757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:11:43.345705 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2757
I20260812 00:11:43.345741 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.336300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:57609 (local address 127.31.227.62:38649)
0812 00:11:43.336369 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.336371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.336660 (+ 289us) server_negotiation.cc:407] Connection header received
0812 00:11:43.337385 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.337386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.337398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.337447 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.338390 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.338874 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.339897 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.340098 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.340189 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.342513 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.342515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.342516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.342524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.344302 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.344769 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.344772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.344772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.344846 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.345150 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.345151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.345151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.345262 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.345321 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.345594 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.345676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:11:43.345806 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:43.345964 32652 external_mini_cluster.cc:1428] Setting key 45a17d83abbc6dbe7e98d0ca30ec50d1
I20260812 00:11:43.346006 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.336277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.31.227.3:57609)
0812 00:11:43.336487 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.336540 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.337277 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.337550 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.337555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.337951 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:43.338253 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.338259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.338984 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.338985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.339786 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.339788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.340221 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.340222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.340362 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.340845 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:43.340858 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:43.342411 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:43.344386 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.344392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.344393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.344684 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.344957 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.344958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.344959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.345100 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.345325 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:43.345330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:43.345440 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.345578 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.345755 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
I20260812 00:11:43.346465 2968 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38649
I20260812 00:11:43.346556 2968 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:43.346738 2968 heartbeater.cc:511] Master 127.31.227.62:38649 requested a full tablet report, sending...
I20260812 00:11:43.347093 2129 ts_manager.cc:194] Registered new tserver with Master: 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497)
I20260812 00:11:43.347469 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:57609
I20260812 00:11:43.352377 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.347657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:57371 (local address 127.31.227.62:38649)
0812 00:11:43.347700 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.347703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.347712 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:11:43.347764 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.347765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.347776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.347830 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.348168 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.348604 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.349254 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.349428 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.349483 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.350417 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.350418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.350418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.350424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.351540 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.351836 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.351837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.351837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.351892 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.352042 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.352043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.352043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.352133 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.352181 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.352263 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.352318 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:11:43.352385 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.347537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.31.227.3:57371)
0812 00:11:43.347654 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.347665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.347716 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.347834 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.347834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.348037 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:43.348093 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.348095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.348702 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.348703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.349184 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.349186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.349493 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.349494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.349624 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.350056 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:43.350061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:43.350337 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:43.351613 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.351614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.351614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.351772 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.351949 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:43.351949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:43.351950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:43.351993 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:43.352171 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:43.352172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:43.352204 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.352249 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.352318 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260812 00:11:43.357636 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:11:43 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:43.382091 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.375187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.31.227.62:38649)
0812 00:11:43.375254 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.375257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.375280 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:11:43.375349 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.375350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.375360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.375412 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:43.375969 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.376467 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.377161 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.377338 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.377386 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.379629 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:43.379631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:43.379631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:43.379638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:43.380988 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.381355 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.381356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.381356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.381408 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:43.381626 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:43.381627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:43.381627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:43.381736 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:43.381818 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.381925 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.381986 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:11:43.383864 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:43.385509 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:43.392501 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.388210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:45177 (local address 127.0.0.1:43980)
0812 00:11:43.388447 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.388457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.388495 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.388811 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.388812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.388814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.388875 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.388876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.389951 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.389952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.391171 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.391174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.392182 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.392184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.392264 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.392282 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.392351 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.392411 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":68,"threads_started":1}
I20260812 00:11:43.392619 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.388034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:39497 (local address 127.0.0.1:54742)
0812 00:11:43.388339 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.388351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.388406 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.389138 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.389139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.389142 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.389183 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.389185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.390474 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.390475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.391549 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.391552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.392341 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.392342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.392412 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.392416 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.392458 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.392487 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
I20260812 00:11:43.392827 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.388017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35781 (local address 127.0.0.1:47218)
0812 00:11:43.388209 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.388255 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.388324 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.388715 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.388719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.388721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.388797 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.388799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.390222 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.390223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.391199 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.391202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.392268 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.392270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.392376 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.392382 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.392436 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.392482 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260812 00:11:43.392905 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.388140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.31.227.3:39497)
0812 00:11:43.388964 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.388970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.388980 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:43.389021 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.389023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.389037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.389259 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.389321 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.390374 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.391631 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.392256 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.392334 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.392452 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.392521 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.392567 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":99,"threads_started":1}
I20260812 00:11:43.392967 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.388470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.31.227.2:45177)
0812 00:11:43.388664 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.388668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.388675 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:11:43.388708 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.388709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.388721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.388813 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.389024 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.389857 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.391264 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.392097 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.392169 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.392230 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.392718 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.392766 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":111,"threads_started":1}
I20260812 00:11:43.393076 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.388138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.31.227.1:35781)
0812 00:11:43.388535 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.388541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.388552 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:11:43.388599 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.388601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.388618 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.388694 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.388897 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.390109 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.391314 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.392092 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.392168 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.392230 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.392560 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.392775 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":107,"threads_started":1}
I20260812 00:11:43.393641 2868 tablet_service.cc:1467] Processing CreateTablet for tablet ec1e604a07554a488e78c26e45894b9b (DEFAULT_TABLE table=test-table [id=6b68faf283334304b2159d0de02da232]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:43.393661 2616 tablet_service.cc:1467] Processing CreateTablet for tablet ec1e604a07554a488e78c26e45894b9b (DEFAULT_TABLE table=test-table [id=6b68faf283334304b2159d0de02da232]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:43.394017 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e604a07554a488e78c26e45894b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:43.394037 2616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e604a07554a488e78c26e45894b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:43.393939 2370 tablet_service.cc:1467] Processing CreateTablet for tablet ec1e604a07554a488e78c26e45894b9b (DEFAULT_TABLE table=test-table [id=6b68faf283334304b2159d0de02da232]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:43.394253 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e604a07554a488e78c26e45894b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:43.397004 3014 tablet_bootstrap.cc:492] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Bootstrap starting.
I20260812 00:11:43.397869 3017 tablet_bootstrap.cc:492] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Bootstrap starting.
I20260812 00:11:43.398138 3014 tablet_bootstrap.cc:654] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:43.398524 3014 log.cc:824] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:43.398579 3017 tablet_bootstrap.cc:654] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:43.398903 3017 log.cc:824] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:43.399225 3016 tablet_bootstrap.cc:492] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Bootstrap starting.
I20260812 00:11:43.399425 3014 tablet_bootstrap.cc:492] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: No bootstrap required, opened a new log
I20260812 00:11:43.399483 3014 ts_tablet_manager.cc:1397] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 00:11:43.399916 3016 tablet_bootstrap.cc:654] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:43.400230 3016 log.cc:824] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:43.400820 3017 tablet_bootstrap.cc:492] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: No bootstrap required, opened a new log
I20260812 00:11:43.400904 3017 ts_tablet_manager.cc:1397] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:11:43.401417 3014 raft_consensus.cc:348] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.401608 3014 raft_consensus.cc:374] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:43.401655 3014 raft_consensus.cc:729] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492c8da422c04d3d8590796b752bd4bc, State: Initialized, Role: FOLLOWER
I20260812 00:11:43.401786 3014 consensus_queue.cc:260] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.401921 3016 tablet_bootstrap.cc:492] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: No bootstrap required, opened a new log
I20260812 00:11:43.401991 3016 ts_tablet_manager.cc:1397] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:11:43.402063 3014 ts_tablet_manager.cc:1428] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 00:11:43.402179 2968 heartbeater.cc:503] Master 127.31.227.62:38649 was elected leader, sending a full tablet report...
I20260812 00:11:43.402976 3017 raft_consensus.cc:348] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.403139 3017 raft_consensus.cc:374] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:43.403182 3017 raft_consensus.cc:729] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74e637f47b947beacdbdeded61349b8, State: Initialized, Role: FOLLOWER
I20260812 00:11:43.403313 3017 consensus_queue.cc:260] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.403597 3017 ts_tablet_manager.cc:1428] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:11:43.403645 2743 heartbeater.cc:503] Master 127.31.227.62:38649 was elected leader, sending a full tablet report...
I20260812 00:11:43.404170 3016 raft_consensus.cc:348] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.404321 3016 raft_consensus.cc:374] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:43.404345 3016 raft_consensus.cc:729] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdd1df89807483c89f82d22f0ed03dc, State: Initialized, Role: FOLLOWER
I20260812 00:11:43.404469 3016 consensus_queue.cc:260] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.404737 3016 ts_tablet_manager.cc:1428] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:11:43.404811 2482 heartbeater.cc:503] Master 127.31.227.62:38649 was elected leader, sending a full tablet report...
W20260812 00:11:43.439299 2745 tablet.cc:2369] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:11:43.532147 2484 tablet.cc:2369] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:11:43.595335 2971 tablet.cc:2369] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:11:43.738229 3024 raft_consensus.cc:482] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:43.738410 3024 raft_consensus.cc:504] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.738830 3024 leader_election.cc:288] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781), 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497)
I20260812 00:11:43.742172 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.739164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34865 (local address 127.31.227.1:35781)
0812 00:11:43.739219 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.739221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.739412 (+ 191us) server_negotiation.cc:407] Connection header received
0812 00:11:43.739472 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.739474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.739481 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.739546 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.739719 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.740243 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.741400 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.741783 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.741884 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.741893 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.742052 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.742095 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:11:43.742189 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.739154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:49195 (local address 127.31.227.3:39497)
0812 00:11:43.739218 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.739221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.739361 (+ 140us) server_negotiation.cc:407] Connection header received
0812 00:11:43.739470 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.739471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.739480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.739547 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.739752 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.740227 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.741388 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.741817 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.741898 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.741906 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.742082 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.742117 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:11:43.742219 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.739055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:39497 (local address 127.31.227.2:49195)
0812 00:11:43.739268 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.739278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.739358 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.739557 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.739558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.739561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.739650 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.739652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.740407 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.740408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.741290 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.741293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.741912 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.741913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.741998 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.742002 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.742062 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.742106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260812 00:11:43.742412 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.739055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35781 (local address 127.31.227.2:34865)
0812 00:11:43.739304 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.739312 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.739401 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.739552 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.739553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.739555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.739641 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.739643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.740345 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.740346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.741292 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.741295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.741890 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.741891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.741971 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.741975 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.742045 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.742089 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260812 00:11:43.742532 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e604a07554a488e78c26e45894b9b" candidate_uuid: "a74e637f47b947beacdbdeded61349b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492c8da422c04d3d8590796b752bd4bc" is_pre_election: true
I20260812 00:11:43.742709 2896 raft_consensus.cc:2454] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74e637f47b947beacdbdeded61349b8 in term 0.
I20260812 00:11:43.742797 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e604a07554a488e78c26e45894b9b" candidate_uuid: "a74e637f47b947beacdbdeded61349b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdd1df89807483c89f82d22f0ed03dc" is_pre_election: true
I20260812 00:11:43.742944 2407 raft_consensus.cc:2454] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74e637f47b947beacdbdeded61349b8 in term 0.
I20260812 00:11:43.742962 2537 leader_election.cc:302] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492c8da422c04d3d8590796b752bd4bc, a74e637f47b947beacdbdeded61349b8; no voters:
I20260812 00:11:43.743098 3024 raft_consensus.cc:2781] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:43.743163 3024 raft_consensus.cc:482] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:43.743180 3024 raft_consensus.cc:3037] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:43.743885 3024 raft_consensus.cc:504] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.744024 3024 leader_election.cc:288] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [CANDIDATE]: Term 1 election: Requested vote from peers dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781), 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497)
I20260812 00:11:43.744215 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e604a07554a488e78c26e45894b9b" candidate_uuid: "a74e637f47b947beacdbdeded61349b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492c8da422c04d3d8590796b752bd4bc"
I20260812 00:11:43.744258 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e604a07554a488e78c26e45894b9b" candidate_uuid: "a74e637f47b947beacdbdeded61349b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdd1df89807483c89f82d22f0ed03dc"
I20260812 00:11:43.744298 2896 raft_consensus.cc:3037] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:43.744349 2407 raft_consensus.cc:3037] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:43.745046 2896 raft_consensus.cc:2454] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74e637f47b947beacdbdeded61349b8 in term 1.
I20260812 00:11:43.745054 2407 raft_consensus.cc:2454] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74e637f47b947beacdbdeded61349b8 in term 1.
I20260812 00:11:43.745239 2537 leader_election.cc:302] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492c8da422c04d3d8590796b752bd4bc, a74e637f47b947beacdbdeded61349b8; no voters:
I20260812 00:11:43.745344 3024 raft_consensus.cc:2781] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:43.745479 3024 raft_consensus.cc:686] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 1 LEADER]: Becoming Leader. State: Replica: a74e637f47b947beacdbdeded61349b8, State: Running, Role: LEADER
I20260812 00:11:43.745599 3024 consensus_queue.cc:237] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.746381 2129 catalog_manager.cc:5495] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74e637f47b947beacdbdeded61349b8 (127.31.227.2). New cstate: current_term: 1 leader_uuid: "a74e637f47b947beacdbdeded61349b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } health_report { overall_health: UNKNOWN } } }
I20260812 00:11:43.833441 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.830918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.31.227.2:45177)
0812 00:11:43.830984 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.830986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.830995 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:11:43.831089 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.831090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.831108 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.831167 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:43.831304 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.831771 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.832551 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.832763 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.832828 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.833016 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:43.833111 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:43.833173 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.833282 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.833303 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:11:43.835436 2407 raft_consensus.cc:1261] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Refusing update from remote peer a74e637f47b947beacdbdeded61349b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:43.835445 2896 raft_consensus.cc:1261] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Refusing update from remote peer a74e637f47b947beacdbdeded61349b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:43.835798 3024 consensus_queue.cc:1035] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:43.835961 3030 consensus_queue.cc:1035] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:43.838725 3033 mvcc.cc:205] Tried to move back new op lower bound from 7317477391707156480 to 7317477391342678016. Current Snapshot: MvccSnapshot[applied={T|T < 7317477391707156480}]
I20260812 00:11:43.838846 3036 mvcc.cc:205] Tried to move back new op lower bound from 7317477391707156480 to 7317477391342678016. Current Snapshot: MvccSnapshot[applied={T|T < 7317477391707156480}]
I20260812 00:11:43.840149 3035 mvcc.cc:205] Tried to move back new op lower bound from 7317477391707156480 to 7317477391342678016. Current Snapshot: MvccSnapshot[applied={T|T < 7317477391707156480 or (T in {7317477391707156480})}]
I20260812 00:11:43.842154 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:43.842262 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:43.843194 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: Sending DeleteTablet for 3 replicas of tablet ec1e604a07554a488e78c26e45894b9b
I20260812 00:11:43.843557 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1e604a07554a488e78c26e45894b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:43 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:47218
I20260812 00:11:43.843544 2868 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1e604a07554a488e78c26e45894b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:43 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:54742
I20260812 00:11:43.843770 3051 tablet_replica.cc:330] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: stopping tablet replica
I20260812 00:11:43.843815 2616 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1e604a07554a488e78c26e45894b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:43 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:43980
I20260812 00:11:43.843799 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 00:11:43.843875 3051 raft_consensus.cc:2229] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 00:11:43.843938 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:43.843978 3051 raft_consensus.cc:2258] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:43.844017 3052 tablet_replica.cc:330] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: stopping tablet replica
I20260812 00:11:43.844110 3052 raft_consensus.cc:2229] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:11:43.844399 3050 tablet_replica.cc:330] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: stopping tablet replica
I20260812 00:11:43.844501 3050 raft_consensus.cc:2229] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:43.844602 3050 raft_consensus.cc:2258] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:43.844733 3052 raft_consensus.cc:2258] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:43.845208 2616 tablet_service.cc:1467] Processing CreateTablet for tablet 945a8ba8c044491f94f6864ab2d5a10d (DEFAULT_TABLE table=test-table [id=79ff73009a4d4fb08fd55acb7ea2451f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:43.845310 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 945a8ba8c044491f94f6864ab2d5a10d (DEFAULT_TABLE table=test-table [id=79ff73009a4d4fb08fd55acb7ea2451f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:43.845348 2616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945a8ba8c044491f94f6864ab2d5a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:43.845432 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945a8ba8c044491f94f6864ab2d5a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:43.845589 3050 ts_tablet_manager.cc:1905] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:43.846889 3016 tablet_bootstrap.cc:492] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Bootstrap starting.
I20260812 00:11:43.846966 2868 tablet_service.cc:1467] Processing CreateTablet for tablet 945a8ba8c044491f94f6864ab2d5a10d (DEFAULT_TABLE table=test-table [id=79ff73009a4d4fb08fd55acb7ea2451f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:43.847095 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945a8ba8c044491f94f6864ab2d5a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:43.847298 3016 tablet_bootstrap.cc:654] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:43.845352 3052 ts_tablet_manager.cc:1905] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:43.847784 3050 ts_tablet_manager.cc:1918] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:11:43.847862 3050 log.cc:1196] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/ec1e604a07554a488e78c26e45894b9b
I20260812 00:11:43.848135 3050 ts_tablet_manager.cc:1939] T ec1e604a07554a488e78c26e45894b9b P dbdd1df89807483c89f82d22f0ed03dc: Deleting consensus metadata
I20260812 00:11:43.848595 3014 tablet_bootstrap.cc:492] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Bootstrap starting.
I20260812 00:11:43.848639 3017 tablet_bootstrap.cc:492] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Bootstrap starting.
I20260812 00:11:43.849035 3014 tablet_bootstrap.cc:654] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:43.849107 3017 tablet_bootstrap.cc:654] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:43.849697 2110 catalog_manager.cc:4841] TS dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781): tablet ec1e604a07554a488e78c26e45894b9b (table test-table [id=6b68faf283334304b2159d0de02da232]) successfully deleted
I20260812 00:11:43.849763 3052 ts_tablet_manager.cc:1918] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:11:43.849844 3052 log.cc:1196] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/ec1e604a07554a488e78c26e45894b9b
I20260812 00:11:43.850085 3051 ts_tablet_manager.cc:1905] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:43.850119 3052 ts_tablet_manager.cc:1939] T ec1e604a07554a488e78c26e45894b9b P a74e637f47b947beacdbdeded61349b8: Deleting consensus metadata
I20260812 00:11:43.850545 2110 catalog_manager.cc:4841] TS a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177): tablet ec1e604a07554a488e78c26e45894b9b (table test-table [id=6b68faf283334304b2159d0de02da232]) successfully deleted
I20260812 00:11:43.851478 3051 ts_tablet_manager.cc:1918] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:11:43.851553 3051 log.cc:1196] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/ec1e604a07554a488e78c26e45894b9b
I20260812 00:11:43.851701 3017 tablet_bootstrap.cc:492] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: No bootstrap required, opened a new log
I20260812 00:11:43.851749 3017 ts_tablet_manager.cc:1397] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:11:43.851931 3017 raft_consensus.cc:348] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.851963 3051 ts_tablet_manager.cc:1939] T ec1e604a07554a488e78c26e45894b9b P 492c8da422c04d3d8590796b752bd4bc: Deleting consensus metadata
I20260812 00:11:43.852015 3017 raft_consensus.cc:374] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:43.852046 3017 raft_consensus.cc:729] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74e637f47b947beacdbdeded61349b8, State: Initialized, Role: FOLLOWER
I20260812 00:11:43.852085 3017 consensus_queue.cc:260] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.852182 3017 ts_tablet_manager.cc:1428] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:43.852384 2107 catalog_manager.cc:4841] TS 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497): tablet ec1e604a07554a488e78c26e45894b9b (table test-table [id=6b68faf283334304b2159d0de02da232]) successfully deleted
I20260812 00:11:43.852892 3016 tablet_bootstrap.cc:492] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: No bootstrap required, opened a new log
I20260812 00:11:43.852967 3016 ts_tablet_manager.cc:1397] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260812 00:11:43.853276 3016 raft_consensus.cc:348] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.853371 3016 raft_consensus.cc:374] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:43.853405 3016 raft_consensus.cc:729] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdd1df89807483c89f82d22f0ed03dc, State: Initialized, Role: FOLLOWER
I20260812 00:11:43.853457 3016 consensus_queue.cc:260] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.853588 3016 ts_tablet_manager.cc:1428] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:43.853935 3014 tablet_bootstrap.cc:492] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: No bootstrap required, opened a new log
I20260812 00:11:43.854009 3014 ts_tablet_manager.cc:1397] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 00:11:43.854142 3014 raft_consensus.cc:348] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.854235 3014 raft_consensus.cc:374] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:43.854266 3014 raft_consensus.cc:729] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492c8da422c04d3d8590796b752bd4bc, State: Initialized, Role: FOLLOWER
I20260812 00:11:43.854316 3014 consensus_queue.cc:260] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.854419 3014 ts_tablet_manager.cc:1428] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:43.961226 3025 raft_consensus.cc:482] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:43.961349 3025 raft_consensus.cc:504] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.961766 3025 leader_election.cc:288] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177), 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497)
I20260812 00:11:43.965925 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.962014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:52079 (local address 127.31.227.2:45177)
0812 00:11:43.962073 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.962076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.962193 (+ 117us) server_negotiation.cc:407] Connection header received
0812 00:11:43.962253 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.962254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.962266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.962340 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.962556 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.963302 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.964943 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.965498 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.965600 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.965610 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.965807 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.965847 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:11:43.965919 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.962086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:33881 (local address 127.31.227.3:39497)
0812 00:11:43.962131 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:43.962133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:43.962203 (+ 70us) server_negotiation.cc:407] Connection header received
0812 00:11:43.962255 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:43.962255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:43.962265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:43.962340 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:43.962556 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.963302 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.964941 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.965543 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.965606 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.965613 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:43.965807 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:11:43.965850 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:11:43.965962 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.961955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:39497 (local address 127.31.227.1:33881)
0812 00:11:43.962125 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.962136 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.962191 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.962356 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.962357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.962358 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.962481 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.962482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.963410 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.963411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.964835 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.964839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.965632 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.965633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.965721 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.965728 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.965787 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.965852 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":116,"thread_start_us":99,"threads_started":1}
I20260812 00:11:43.965963 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:43.961924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:45177 (local address 127.31.227.1:52079)
0812 00:11:43.962103 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:43.962117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:43.962192 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:43.962356 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:43.962357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:43.962360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:43.962461 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.962463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.963394 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.963395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.964835 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:43.964839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:43.965632 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:43.965633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:43.965721 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:43.965729 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 00:11:43.965787 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 00:11:43.965852 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":80,"thread_start_us":68,"threads_started":1}
I20260812 00:11:43.966293 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945a8ba8c044491f94f6864ab2d5a10d" candidate_uuid: "dbdd1df89807483c89f82d22f0ed03dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492c8da422c04d3d8590796b752bd4bc" is_pre_election: true
I20260812 00:11:43.966411 2896 raft_consensus.cc:2454] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbdd1df89807483c89f82d22f0ed03dc in term 0.
I20260812 00:11:43.966509 2656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945a8ba8c044491f94f6864ab2d5a10d" candidate_uuid: "dbdd1df89807483c89f82d22f0ed03dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74e637f47b947beacdbdeded61349b8" is_pre_election: true
I20260812 00:11:43.966635 2656 raft_consensus.cc:2454] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbdd1df89807483c89f82d22f0ed03dc in term 0.
I20260812 00:11:43.966621 2289 leader_election.cc:302] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492c8da422c04d3d8590796b752bd4bc, dbdd1df89807483c89f82d22f0ed03dc; no voters:
I20260812 00:11:43.966780 3025 raft_consensus.cc:2781] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:43.966830 3025 raft_consensus.cc:482] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:43.966861 3025 raft_consensus.cc:3037] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:43.967622 3025 raft_consensus.cc:504] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.967800 3025 leader_election.cc:288] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [CANDIDATE]: Term 1 election: Requested vote from peers a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177), 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497)
I20260812 00:11:43.967940 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945a8ba8c044491f94f6864ab2d5a10d" candidate_uuid: "dbdd1df89807483c89f82d22f0ed03dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492c8da422c04d3d8590796b752bd4bc"
I20260812 00:11:43.968021 2896 raft_consensus.cc:3037] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:43.968633 2896 raft_consensus.cc:2454] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbdd1df89807483c89f82d22f0ed03dc in term 1.
I20260812 00:11:43.968757 2656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945a8ba8c044491f94f6864ab2d5a10d" candidate_uuid: "dbdd1df89807483c89f82d22f0ed03dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74e637f47b947beacdbdeded61349b8"
I20260812 00:11:43.968782 2289 leader_election.cc:302] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492c8da422c04d3d8590796b752bd4bc, dbdd1df89807483c89f82d22f0ed03dc; no voters:
I20260812 00:11:43.968837 2656 raft_consensus.cc:3037] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:43.968924 3025 raft_consensus.cc:2781] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:43.969086 3025 raft_consensus.cc:686] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 1 LEADER]: Becoming Leader. State: Replica: dbdd1df89807483c89f82d22f0ed03dc, State: Running, Role: LEADER
I20260812 00:11:43.969158 3025 consensus_queue.cc:237] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:43.969754 2656 raft_consensus.cc:2454] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbdd1df89807483c89f82d22f0ed03dc in term 1.
I20260812 00:11:43.969890 2128 catalog_manager.cc:5495] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc reported cstate change: term changed from 0 to 1, leader changed from <none> to dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1). New cstate: current_term: 1 leader_uuid: "dbdd1df89807483c89f82d22f0ed03dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:11:44 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493502, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:44.013760 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.006308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.31.227.62:38649)
0812 00:11:44.006468 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.006471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.006547 (+ 76us) server_negotiation.cc:407] Connection header received
0812 00:11:44.006658 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.006659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.006672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.006769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:44.007237 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.007814 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.008531 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.008742 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.008827 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.011006 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:44.011007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:44.011007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:44.011014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:44.012439 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:44.012837 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:44.012838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:44.012839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:44.012919 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:44.013124 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:44.013126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:44.013126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:44.013239 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:44.013310 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.013478 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.013635 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260812 00:11:44.013818 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.006243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38649 (local address 127.0.0.1:36804)
0812 00:11:44.006458 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:44.006471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:44.006586 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:44.006773 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:44.006774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:44.007103 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:44.007151 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.007153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.007903 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.007905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.008447 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.008449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.008828 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.008829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.008880 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.009460 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:44.009465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:44.010918 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:44.012538 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:44.012546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:44.012549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:44.012745 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:44.012986 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:44.012987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:44.012987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:44.013048 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:44.013312 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:44.013321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:44.013381 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 00:11:44.013453 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:11:44.013694 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":137,"thread_start_us":125,"threads_started":1}
I20260812 00:11:44.016040 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:36804:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:11:44.016317 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:44.018090 2370 tablet_service.cc:1467] Processing CreateTablet for tablet b68f8eedfa664f2c838a1ce1985a6151 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8db2a3e77f1640cd9bae1ca4058f2eef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:11:44.018100 2868 tablet_service.cc:1467] Processing CreateTablet for tablet b68f8eedfa664f2c838a1ce1985a6151 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8db2a3e77f1640cd9bae1ca4058f2eef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:11:44.018270 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68f8eedfa664f2c838a1ce1985a6151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:44.018275 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68f8eedfa664f2c838a1ce1985a6151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:44.019593 3014 tablet_bootstrap.cc:492] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: Bootstrap starting.
I20260812 00:11:44.019606 3016 tablet_bootstrap.cc:492] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: Bootstrap starting.
I20260812 00:11:44.019735 2616 tablet_service.cc:1467] Processing CreateTablet for tablet b68f8eedfa664f2c838a1ce1985a6151 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8db2a3e77f1640cd9bae1ca4058f2eef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:11:44.019898 2616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68f8eedfa664f2c838a1ce1985a6151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:44.020099 3014 tablet_bootstrap.cc:654] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:44.020109 3016 tablet_bootstrap.cc:654] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:44.020808 3016 tablet_bootstrap.cc:492] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: No bootstrap required, opened a new log
I20260812 00:11:44.020872 3016 ts_tablet_manager.cc:1397] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:44.021004 3014 tablet_bootstrap.cc:492] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: No bootstrap required, opened a new log
I20260812 00:11:44.021059 3014 ts_tablet_manager.cc:1397] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:44.021035 3016 raft_consensus.cc:348] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.021142 3016 raft_consensus.cc:374] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:44.021181 3016 raft_consensus.cc:729] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdd1df89807483c89f82d22f0ed03dc, State: Initialized, Role: FOLLOWER
I20260812 00:11:44.021230 3016 consensus_queue.cc:260] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.021281 3014 raft_consensus.cc:348] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.021380 3014 raft_consensus.cc:374] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:44.021381 3016 ts_tablet_manager.cc:1428] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:44.021416 3014 raft_consensus.cc:729] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492c8da422c04d3d8590796b752bd4bc, State: Initialized, Role: FOLLOWER
I20260812 00:11:44.021473 3014 consensus_queue.cc:260] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.021585 3014 ts_tablet_manager.cc:1428] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:44.021582 3025 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.021667 3025 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.021833 3023 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.021937 3023 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.022013 3017 tablet_bootstrap.cc:492] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: Bootstrap starting.
I20260812 00:11:44.022269 3017 tablet_bootstrap.cc:654] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:44.022845 3017 tablet_bootstrap.cc:492] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: No bootstrap required, opened a new log
I20260812 00:11:44.022908 3017 ts_tablet_manager.cc:1397] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 00:11:44.023025 3017 raft_consensus.cc:348] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.023095 3017 raft_consensus.cc:374] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:44.023110 3017 raft_consensus.cc:729] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74e637f47b947beacdbdeded61349b8, State: Initialized, Role: FOLLOWER
I20260812 00:11:44.023164 3017 consensus_queue.cc:260] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.023260 3017 ts_tablet_manager.cc:1428] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:44.023290 3042 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.023396 3042 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.048362 3023 raft_consensus.cc:482] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:44.048439 3023 raft_consensus.cc:504] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.048733 3023 leader_election.cc:288] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781), a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177)
I20260812 00:11:44.052587 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.049220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:49043 (local address 127.31.227.2:45177)
0812 00:11:44.049262 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.049264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.049295 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:11:44.049387 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.049388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.049411 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.049472 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:44.049689 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.050376 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.051679 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.052203 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.052280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.052290 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.052482 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.052521 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:11:44.052587 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.048953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:37595 (local address 127.31.227.1:35781)
0812 00:11:44.049022 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.049024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.049109 (+ 85us) server_negotiation.cc:407] Connection header received
0812 00:11:44.049176 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.049177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.049188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.049252 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:44.049436 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.050227 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.051681 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.052225 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.052285 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.052292 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.052482 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.052522 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:11:44.052639 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.049150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:45177 (local address 127.31.227.3:49043)
0812 00:11:44.049238 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:44.049247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:44.049304 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:44.049502 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:44.049503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:44.049504 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:44.049602 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.049604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.050453 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.050453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.051591 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.051594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.052344 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.052345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.052409 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.052412 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:44.052461 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:11:44.052521 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":13,"thread_start_us":58,"threads_started":1}
I20260812 00:11:44.052647 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.048883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35781 (local address 127.31.227.3:37595)
0812 00:11:44.049016 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:44.049027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:44.049115 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:44.049274 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:44.049275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:44.049277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:44.049364 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.049367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.050312 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.050313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.051591 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.051594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.052322 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.052323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.052394 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.052399 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:11:44.052461 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 00:11:44.052521 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":14,"thread_start_us":88,"threads_started":1}
I20260812 00:11:44.052937 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68f8eedfa664f2c838a1ce1985a6151" candidate_uuid: "492c8da422c04d3d8590796b752bd4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdd1df89807483c89f82d22f0ed03dc" is_pre_election: true
I20260812 00:11:44.053030 2407 raft_consensus.cc:2454] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492c8da422c04d3d8590796b752bd4bc in term 0.
I20260812 00:11:44.053071 2656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68f8eedfa664f2c838a1ce1985a6151" candidate_uuid: "492c8da422c04d3d8590796b752bd4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74e637f47b947beacdbdeded61349b8" is_pre_election: true
I20260812 00:11:44.053172 2656 raft_consensus.cc:2454] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492c8da422c04d3d8590796b752bd4bc in term 0.
I20260812 00:11:44.053210 2788 leader_election.cc:302] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492c8da422c04d3d8590796b752bd4bc, dbdd1df89807483c89f82d22f0ed03dc; no voters:
I20260812 00:11:44.053347 3023 raft_consensus.cc:2781] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:44.053388 3023 raft_consensus.cc:482] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:44.053412 3023 raft_consensus.cc:3037] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:44.054126 3023 raft_consensus.cc:504] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.054260 3023 leader_election.cc:288] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [CANDIDATE]: Term 1 election: Requested vote from peers dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781), a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177)
I20260812 00:11:44.054488 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68f8eedfa664f2c838a1ce1985a6151" candidate_uuid: "492c8da422c04d3d8590796b752bd4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdd1df89807483c89f82d22f0ed03dc"
I20260812 00:11:44.054488 2656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68f8eedfa664f2c838a1ce1985a6151" candidate_uuid: "492c8da422c04d3d8590796b752bd4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74e637f47b947beacdbdeded61349b8"
I20260812 00:11:44.054571 2407 raft_consensus.cc:3037] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:44.054572 2656 raft_consensus.cc:3037] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:44.055310 2407 raft_consensus.cc:2454] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492c8da422c04d3d8590796b752bd4bc in term 1.
I20260812 00:11:44.055310 2656 raft_consensus.cc:2454] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492c8da422c04d3d8590796b752bd4bc in term 1.
I20260812 00:11:44.055483 2788 leader_election.cc:302] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492c8da422c04d3d8590796b752bd4bc, dbdd1df89807483c89f82d22f0ed03dc; no voters:
I20260812 00:11:44.055593 3023 raft_consensus.cc:2781] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:44.055716 3023 raft_consensus.cc:686] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [term 1 LEADER]: Becoming Leader. State: Replica: 492c8da422c04d3d8590796b752bd4bc, State: Running, Role: LEADER
I20260812 00:11:44.055814 3023 consensus_queue.cc:237] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } }
I20260812 00:11:44.056177 3084 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: TxnStatusTablet state changed. Reason: New leader 492c8da422c04d3d8590796b752bd4bc. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.056267 3084 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:11:44.056524 3088 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:11:44.056625 3088 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:11:44.056547 2128 catalog_manager.cc:5495] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 492c8da422c04d3d8590796b752bd4bc (127.31.227.3). New cstate: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } health_report { overall_health: HEALTHY } } }
I20260812 00:11:44.070116 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.067676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.31.227.3:39497)
0812 00:11:44.067734 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.067736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.067742 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:11:44.067769 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.067770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.067784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.067873 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:44.068029 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.068489 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.069257 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.069451 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.069573 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.069785 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.069888 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.069976 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.070032 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.070052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:11:44.070158 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.067268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:39497 (local address 127.0.0.1:54744)
0812 00:11:44.067356 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:44.067368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:44.067418 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:44.067883 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:44.067885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:44.067886 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:44.067959 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.067961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.068587 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.068588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.069172 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.069174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.069553 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.069554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.069622 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.069637 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.069960 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.069971 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 00:11:44.070028 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:11:44.070080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:11:44.071412 2407 raft_consensus.cc:1261] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Refusing update from remote peer 492c8da422c04d3d8590796b752bd4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:44.071444 2656 raft_consensus.cc:1261] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Refusing update from remote peer 492c8da422c04d3d8590796b752bd4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:44.071655 3084 consensus_queue.cc:1035] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:44.071766 3023 consensus_queue.cc:1035] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:44.072415 3042 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: TxnStatusTablet state changed. Reason: New leader 492c8da422c04d3d8590796b752bd4bc. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.072517 3042 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.072577 3025 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: TxnStatusTablet state changed. Reason: New leader 492c8da422c04d3d8590796b752bd4bc. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.072657 3025 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.073009 3084 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.073094 3084 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:11:44.074050 3025 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.074141 3025 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P dbdd1df89807483c89f82d22f0ed03dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.075039 3084 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.075136 3084 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P 492c8da422c04d3d8590796b752bd4bc: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:11:44.075790 3042 tablet_replica.cc:439] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "492c8da422c04d3d8590796b752bd4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdd1df89807483c89f82d22f0ed03dc" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35781 } } peers { permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 } } peers { permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 } } }
I20260812 00:11:44.075883 3042 tablet_replica.cc:442] T b68f8eedfa664f2c838a1ce1985a6151 P a74e637f47b947beacdbdeded61349b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:44.078267 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.075446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.31.227.1:35781)
0812 00:11:44.075575 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.075577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.075585 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:11:44.075620 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.075621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.075641 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.075712 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:44.075847 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.076494 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.077572 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.077765 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.077815 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.077990 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.078071 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.078121 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.078188 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.078203 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 00:11:44.083534 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.080680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38295 (local address 127.31.227.3:39497)
0812 00:11:44.080737 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.080738 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.080920 (+ 182us) server_negotiation.cc:407] Connection header received
0812 00:11:44.081041 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.081042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.081052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.081102 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:44.081596 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.082036 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.082797 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.083000 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.083075 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.083256 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.083306 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.083376 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.083438 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.083455 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:11:44.083568 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.080614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:39497 (local address 127.31.227.1:38295)
0812 00:11:44.080806 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:44.080855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:44.080934 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:44.081104 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:44.081105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:44.081106 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:44.081466 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.081468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.082120 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.082121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.082695 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.082697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.083079 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.083080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.083145 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.083159 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.083377 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.083380 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:44.083434 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:11:44.083474 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260812 00:11:44.085969 2656 raft_consensus.cc:1261] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Refusing update from remote peer dbdd1df89807483c89f82d22f0ed03dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:44.085985 2896 raft_consensus.cc:1261] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Refusing update from remote peer dbdd1df89807483c89f82d22f0ed03dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:44.086236 3063 consensus_queue.cc:1035] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "492c8da422c04d3d8590796b752bd4bc" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 39497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:44.086392 3025 consensus_queue.cc:1035] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74e637f47b947beacdbdeded61349b8" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:44.094674 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.092111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:45041 (local address 127.31.227.1:35781)
0812 00:11:44.092153 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.092155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.092194 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:11:44.092258 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.092259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.092268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.092321 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:44.092485 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.093119 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.093980 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.094178 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.094253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.094445 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.094477 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.094531 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.094592 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.094611 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:11:44.094955 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.091901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35781 (local address 127.31.227.3:45041)
0812 00:11:44.092135 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:44.092148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:44.092210 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:44.092325 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:44.092326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:44.092343 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:44.092394 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.092396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.093212 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.093213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.093893 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:44.093895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.094254 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.094255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:44.094325 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.094344 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:44.094534 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:44.094537 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:44.094590 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:11:44.094635 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":122,"threads_started":1}
I20260812 00:11:44.105182 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:44.105307 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:44.105988 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84e469f1ca314bd48e9a92bf1c5fea3c: Sending DeleteTablet for 3 replicas of tablet 945a8ba8c044491f94f6864ab2d5a10d
I20260812 00:11:44.106328 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet 945a8ba8c044491f94f6864ab2d5a10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:44 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:47218
I20260812 00:11:44.106428 2616 tablet_service.cc:1514] Processing DeleteTablet for tablet 945a8ba8c044491f94f6864ab2d5a10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:44 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:43980
I20260812 00:11:44.106515 3050 tablet_replica.cc:330] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: stopping tablet replica
I20260812 00:11:44.106599 3050 raft_consensus.cc:2229] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:11:44.106683 3050 raft_consensus.cc:2258] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:44.106752 3052 tablet_replica.cc:330] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: stopping tablet replica
I20260812 00:11:44.106806 3052 raft_consensus.cc:2229] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:44.106839 3052 raft_consensus.cc:2258] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:44.106958 2868 tablet_service.cc:1514] Processing DeleteTablet for tablet 945a8ba8c044491f94f6864ab2d5a10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:44 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:54742
I20260812 00:11:44.107057 3051 tablet_replica.cc:330] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: stopping tablet replica
I20260812 00:11:44.107111 3051 raft_consensus.cc:2229] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:44.107151 3051 raft_consensus.cc:2258] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:44.107223 3052 ts_tablet_manager.cc:1905] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:44.107442 3050 ts_tablet_manager.cc:1905] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:44.107463 3051 ts_tablet_manager.cc:1905] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:44.109009 3051 ts_tablet_manager.cc:1918] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:44.109014 3052 ts_tablet_manager.cc:1918] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:44.109066 3052 log.cc:1196] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/945a8ba8c044491f94f6864ab2d5a10d
I20260812 00:11:44.109066 3051 log.cc:1196] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/945a8ba8c044491f94f6864ab2d5a10d
I20260812 00:11:44.109066 3050 ts_tablet_manager.cc:1918] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:44.109117 3050 log.cc:1196] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/945a8ba8c044491f94f6864ab2d5a10d
I20260812 00:11:44.109300 3052 ts_tablet_manager.cc:1939] T 945a8ba8c044491f94f6864ab2d5a10d P a74e637f47b947beacdbdeded61349b8: Deleting consensus metadata
I20260812 00:11:44.109301 3051 ts_tablet_manager.cc:1939] T 945a8ba8c044491f94f6864ab2d5a10d P 492c8da422c04d3d8590796b752bd4bc: Deleting consensus metadata
I20260812 00:11:44.109346 3050 ts_tablet_manager.cc:1939] T 945a8ba8c044491f94f6864ab2d5a10d P dbdd1df89807483c89f82d22f0ed03dc: Deleting consensus metadata
I20260812 00:11:44.109724 2107 catalog_manager.cc:4841] TS 492c8da422c04d3d8590796b752bd4bc (127.31.227.3:39497): tablet 945a8ba8c044491f94f6864ab2d5a10d (table test-table [id=79ff73009a4d4fb08fd55acb7ea2451f]) successfully deleted
I20260812 00:11:44.109737 2110 catalog_manager.cc:4841] TS a74e637f47b947beacdbdeded61349b8 (127.31.227.2:45177): tablet 945a8ba8c044491f94f6864ab2d5a10d (table test-table [id=79ff73009a4d4fb08fd55acb7ea2451f]) successfully deleted
I20260812 00:11:44.109895 2110 catalog_manager.cc:4841] TS dbdd1df89807483c89f82d22f0ed03dc (127.31.227.1:35781): tablet 945a8ba8c044491f94f6864ab2d5a10d (table test-table [id=79ff73009a4d4fb08fd55acb7ea2451f]) successfully deleted
Aug 12 00:11:44 dist-test-slave-zr1t krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:44.120685 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.110233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.31.227.1:35781)
0812 00:11:44.110276 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.110278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.110408 (+ 130us) server_negotiation.cc:407] Connection header received
0812 00:11:44.110456 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.110457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.110469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.110535 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:44.111260 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.112006 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.113089 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.113400 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.113478 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.117144 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:44.117153 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:44.117153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:44.117161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:44.119297 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:44.119726 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:44.119729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:44.119729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:44.119799 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:44.120061 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:44.120062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:44.120062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:44.120244 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:44.120421 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.120489 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.120576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 00:11:44.120994 2474 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230
I20260812 00:11:44.130074 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:44.122017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.31.227.62:38649)
0812 00:11:44.122071 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:44.122073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:44.122249 (+ 176us) server_negotiation.cc:407] Connection header received
0812 00:11:44.122381 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:44.122383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:44.122395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:44.122428 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:44.123134 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.123948 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.125010 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:44.125348 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:44.125390 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:44.127002 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:44.127004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:44.127004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:44.127012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:44.128746 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:44.129159 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:44.129161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:44.129161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:44.129218 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:44.129418 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:44.129419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:44.129420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:44.129573 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:44.129656 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:44.129878 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 00:11:44.129971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 00:11:44.130708 2129 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812
I20260812 00:11:44.131471 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2255
I20260812 00:11:44.158303 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2500
I20260812 00:11:44.168152 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2757
I20260812 00:11:44.191673 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2080
2026-08-12T00:11:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2918 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:44 dist-test-slave-zr1t krb5kdc[3121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:44 dist-test-slave-zr1t krb5kdc[3121](info): set up 2 sockets
Aug 12 00:11:44 dist-test-slave-zr1t krb5kdc[3121](info): commencing operation
krb5kdc: starting...
W20260812 00:11:45.475092 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:45 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:45.597059 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:33097
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:33097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:45.680791 3139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:45.681000 3139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:45.681023 3139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:45.682134 3139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:45.682169 3139 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:11:45.682183 3139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:45.682195 3139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:45.682209 3139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:45.682220 3139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:45.683653 3139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:33097
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:33097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:45.683844 3139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:45.684015 3139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:45.688952 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:45.688937 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:45.688967 3151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:45.689142 3139 server_base.cc:1034] running on GCE node
I20260812 00:11:45.689515 3139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:45.689819 3139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:45.690968 3139 hybrid_clock.cc:648] HybridClock initialized: now 1786493505690955 us; error 31 us; skew 500 ppm
Aug 12 00:11:45 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493505, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:45.694211 3139 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:45.694784 3139 webserver.cc:460] Webserver started at http://127.31.227.62:36429/ using document root <none> and password file <none>
I20260812 00:11:45.695062 3139 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:45.695134 3139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:45.695277 3139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:45.697144 3139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47"
format_stamp: "Formatted at 2026-08-12 00:11:45 on dist-test-slave-zr1t"
server_key: "c44c226166d02a2302570c7326473ac6"
server_key_iv: "1637dbd3498ede75c2a4b4e491522de3"
server_key_version: "encryptionkey@0"
I20260812 00:11:45.697906 3139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47"
format_stamp: "Formatted at 2026-08-12 00:11:45 on dist-test-slave-zr1t"
server_key: "c44c226166d02a2302570c7326473ac6"
server_key_iv: "1637dbd3498ede75c2a4b4e491522de3"
server_key_version: "encryptionkey@0"
I20260812 00:11:45.704779 3139 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.001s
I20260812 00:11:45.705976 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:45.706195 3139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:45.706321 3139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47"
format_stamp: "Formatted at 2026-08-12 00:11:45 on dist-test-slave-zr1t"
server_key: "c44c226166d02a2302570c7326473ac6"
server_key_iv: "1637dbd3498ede75c2a4b4e491522de3"
server_key_version: "encryptionkey@0"
I20260812 00:11:45.706422 3139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:45.718044 3139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:45.720453 3139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:45.720647 3139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:45.726048 3139 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:33097
I20260812 00:11:45.726081 3275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:33097 every 8 connection(s)
I20260812 00:11:45.726440 3139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:45.727156 3276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:45.729607 3276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Bootstrap starting.
I20260812 00:11:45.730239 3276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:45.730554 3276 log.cc:824] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:45.731266 3276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: No bootstrap required, opened a new log
I20260812 00:11:45.732729 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3139
I20260812 00:11:45.732833 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:45.733062 32652 external_mini_cluster.cc:1428] Setting key ee66084b4cfa0009287d26590c6d10ec
I20260812 00:11:45.734634 3276 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33097 } }
I20260812 00:11:45.734841 3276 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:45.734895 3276 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63eb1ecaaaf4d6bb4d5948cdfce5f47, State: Initialized, Role: FOLLOWER
I20260812 00:11:45.735065 3276 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33097 } }
I20260812 00:11:45.735172 3276 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:45.735220 3276 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:45.735271 3276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:45.735978 3276 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33097 } }
I20260812 00:11:45.736117 3276 leader_election.cc:302] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63eb1ecaaaf4d6bb4d5948cdfce5f47; no voters:
I20260812 00:11:45.736299 3276 leader_election.cc:288] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:45.736330 3283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:45.736442 3283 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [term 1 LEADER]: Becoming Leader. State: Replica: a63eb1ecaaaf4d6bb4d5948cdfce5f47, State: Running, Role: LEADER
I20260812 00:11:45.736536 3283 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33097 } }
I20260812 00:11:45.736573 3276 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:45.736884 3283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63eb1ecaaaf4d6bb4d5948cdfce5f47. Latest consensus state: current_term: 1 leader_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33097 } } }
I20260812 00:11:45.736979 3283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:45.736899 3284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63eb1ecaaaf4d6bb4d5948cdfce5f47" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33097 } } }
I20260812 00:11:45.737186 3284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47 [sys.catalog]: This master's current role is: LEADER
W20260812 00:11:45.738595 3314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:11:45.738651 3314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:11:45.740525 3298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:45.740697 3298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:11:45 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:45.743773 3298 catalog_manager.cc:1347] Generated new cluster ID: d819833e11c040f1a8fb79c960230564
I20260812 00:11:45.743889 3298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:11:45.748756 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:45.733734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.31.227.62:33097)
0812 00:11:45.734055 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:45.734066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:45.734706 (+ 640us) server_negotiation.cc:407] Connection header received
0812 00:11:45.734904 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:45.734913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:45.734978 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:45.735128 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:45.737696 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.739318 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.740430 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.741042 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.741179 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:45.744790 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:45.744794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:45.744794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:45.744808 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:45.747042 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:45.747449 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:45.747453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:45.747453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:45.747587 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:45.747811 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:45.747812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:45.747813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:45.748131 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:45.748288 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:45.748373 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 00:11:45.748531 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260812 00:11:45.751865 3298 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:45.752439 3298 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:45.756712 3298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Generated new TSK 0
I20260812 00:11:45.756883 3298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:45.790211 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33097
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:45.884035 3325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:45.884277 3325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:45.884322 3325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:45.885680 3325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:45.885746 3325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:45.885783 3325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:11:45.885805 3325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:11:45.885838 3325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:45.888548 3325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:45.888839 3325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:45.889110 3325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:45.896100 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:45.896096 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:45.897538 3340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:45.899501 3325 server_base.cc:1034] running on GCE node
I20260812 00:11:45.899739 3325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:45.900076 3325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:45.901279 3325 hybrid_clock.cc:648] HybridClock initialized: now 1786493505901259 us; error 33 us; skew 500 ppm
Aug 12 00:11:45 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493505, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:45.904263 3325 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:45.904903 3325 webserver.cc:460] Webserver started at http://127.31.227.1:46287/ using document root <none> and password file <none>
I20260812 00:11:45.905179 3325 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:45.905258 3325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:45.905391 3325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:45.906610 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "b6435b61b6cc4f61abc372644ef97b13"
format_stamp: "Formatted at 2026-08-12 00:11:45 on dist-test-slave-zr1t"
server_key: "bef85ddd19e26ac39bd961274f4a9e4a"
server_key_iv: "077cdad140307519461077d31ddd1e3a"
server_key_version: "encryptionkey@0"
I20260812 00:11:45.907047 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "b6435b61b6cc4f61abc372644ef97b13"
format_stamp: "Formatted at 2026-08-12 00:11:45 on dist-test-slave-zr1t"
server_key: "bef85ddd19e26ac39bd961274f4a9e4a"
server_key_iv: "077cdad140307519461077d31ddd1e3a"
server_key_version: "encryptionkey@0"
I20260812 00:11:45.908447 3325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:45.909137 3352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:45.909324 3325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:45.909448 3325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "b6435b61b6cc4f61abc372644ef97b13"
format_stamp: "Formatted at 2026-08-12 00:11:45 on dist-test-slave-zr1t"
server_key: "bef85ddd19e26ac39bd961274f4a9e4a"
server_key_iv: "077cdad140307519461077d31ddd1e3a"
server_key_version: "encryptionkey@0"
I20260812 00:11:45.909548 3325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:45.925335 3325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:45.928349 3325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:45.928521 3325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:45.929185 3325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:45.929237 3325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:45.929281 3325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:45.929308 3325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:45.939670 3325 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:44469
I20260812 00:11:45.939741 3551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:44469 every 8 connection(s)
I20260812 00:11:45.940099 3325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:11:45 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493505, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:45.944902 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:45.931739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:58571 (local address 127.31.227.62:33097)
0812 00:11:45.931798 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:45.931800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:45.931815 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:11:45.932252 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:45.932254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:45.932270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:45.932354 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:45.933251 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.934100 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.935675 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.935999 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.936070 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:45.941485 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:45.941487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:45.941488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:45.941498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:45.943420 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:45.943849 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:45.943851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:45.943852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:45.943919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:45.944206 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:45.944207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:45.944208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:45.944337 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:45.944424 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:45.944718 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:11:45.944809 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:11:45.945091 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:45.929772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.31.227.1:58571)
0812 00:11:45.931540 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:45.931583 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:45.932142 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:45.932453 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:45.932458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:45.932842 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:45.933159 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:45.933166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.934217 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.934218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:45.935569 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:45.935572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.936082 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.936083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:45.936235 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:45.939709 (+ 3474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:45.939719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:45.941375 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:45.943519 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:45.943528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:45.943530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:45.943765 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:45.944001 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:45.944002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:45.944002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:45.944124 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:45.944414 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:45.944422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:45.944573 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 00:11:45.944682 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:11:45.944849 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1719,"thread_start_us":1678,"threads_started":1}
I20260812 00:11:45.945732 3552 heartbeater.cc:347] Connected to a master server at 127.31.227.62:33097
I20260812 00:11:45.945848 3552 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:45.946046 3552 heartbeater.cc:511] Master 127.31.227.62:33097 requested a full tablet report, sending...
I20260812 00:11:45.946659 3194 ts_manager.cc:194] Registered new tserver with Master: b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469)
I20260812 00:11:45.946746 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3325
I20260812 00:11:45.946851 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:45.947064 32652 external_mini_cluster.cc:1428] Setting key 94d277f733c840e9b1f34b0d6560b460
I20260812 00:11:45.947517 3194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:58571
I20260812 00:11:45.954286 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:45.946848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:58367 (local address 127.31.227.62:33097)
0812 00:11:45.946891 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:45.946893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:45.947829 (+ 936us) server_negotiation.cc:407] Connection header received
0812 00:11:45.947901 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:45.947901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:45.947912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:45.947959 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:45.948477 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.949183 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.950051 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.950263 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.950349 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:45.951353 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:45.951355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:45.951355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:45.951362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:45.953117 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:45.953487 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:45.953488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:45.953488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:45.953569 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:45.953755 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:45.953756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:45.953756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:45.953872 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:45.953951 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:45.954114 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 00:11:45.954193 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:11:45.954291 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:45.946895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.31.227.1:58367)
0812 00:11:45.947775 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:45.947787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:45.947844 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:45.948026 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:45.948027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:45.948290 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:45.948361 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:45.948364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.949314 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.949315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:45.949963 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:45.949965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:45.950351 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:45.950352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:45.950437 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:45.950895 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:45.950900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:45.951279 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:45.953226 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:45.953227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:45.953227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:45.953409 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:45.953621 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:45.953622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:45.953622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:45.953681 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:45.953949 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:45.953950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:45.953999 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:11:45.954087 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 00:11:45.954195 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:45.979962 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33097
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:46.067451 3573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:46.067675 3573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:46.067700 3573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:46.068791 3573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:46.068823 3573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:46.068843 3573 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:11:46.068854 3573 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:11:46.068871 3573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:46.070307 3573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:46.070493 3573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:46.070670 3573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:46.075806 3582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:46.075793 3587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:46.075794 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:46.075985 3573 server_base.cc:1034] running on GCE node
I20260812 00:11:46.076265 3573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:46.076459 3573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:46.077581 3573 hybrid_clock.cc:648] HybridClock initialized: now 1786493506077562 us; error 31 us; skew 500 ppm
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493506, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:46.080173 3573 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:46.080621 3573 webserver.cc:460] Webserver started at http://127.31.227.2:37987/ using document root <none> and password file <none>
I20260812 00:11:46.080873 3573 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:46.080919 3573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:46.081104 3573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:46.082213 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "ff533b3951cc4dc9b6cc22e05c32daa0"
format_stamp: "Formatted at 2026-08-12 00:11:46 on dist-test-slave-zr1t"
server_key: "92de17fa7aa0493e4c360bbaa64e6973"
server_key_iv: "91fce159840910ddf37ec7e5aee91be1"
server_key_version: "encryptionkey@0"
I20260812 00:11:46.082593 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "ff533b3951cc4dc9b6cc22e05c32daa0"
format_stamp: "Formatted at 2026-08-12 00:11:46 on dist-test-slave-zr1t"
server_key: "92de17fa7aa0493e4c360bbaa64e6973"
server_key_iv: "91fce159840910ddf37ec7e5aee91be1"
server_key_version: "encryptionkey@0"
I20260812 00:11:46.083921 3573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:11:46.084640 3600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:46.084815 3573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:46.084923 3573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "ff533b3951cc4dc9b6cc22e05c32daa0"
format_stamp: "Formatted at 2026-08-12 00:11:46 on dist-test-slave-zr1t"
server_key: "92de17fa7aa0493e4c360bbaa64e6973"
server_key_iv: "91fce159840910ddf37ec7e5aee91be1"
server_key_version: "encryptionkey@0"
I20260812 00:11:46.085005 3573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:46.097602 3573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:46.099391 3573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:46.099556 3573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:46.100054 3573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:46.100119 3573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:46.100181 3573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:46.100210 3573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493506, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:46.108029 3573 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:39303
I20260812 00:11:46.108105 3832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:39303 every 8 connection(s)
I20260812 00:11:46.108399 3573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:46.111141 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.100724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:37911 (local address 127.31.227.62:33097)
0812 00:11:46.100781 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.100783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.101015 (+ 232us) server_negotiation.cc:407] Connection header received
0812 00:11:46.101590 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.101592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.101606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.101671 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:46.102692 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.103338 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.105041 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.105337 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.105441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.108057 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:46.108059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:46.108059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:46.108068 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:46.109736 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.110223 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.110224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.110224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.110279 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.110529 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.110530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.110530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.110640 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:46.110696 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.110952 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.111070 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:11:46.111327 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.100699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.31.227.2:37911)
0812 00:11:46.100882 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.100928 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.101439 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.101787 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.101793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.102206 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:46.102572 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.102581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.103461 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.103462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.104918 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.104921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.105471 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.105472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.105645 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.106377 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:46.106386 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:46.107954 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:46.109866 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.109874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.109876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.110149 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.110346 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.110347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.110348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.110465 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.110703 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:46.110708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:46.110835 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.110922 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.111098 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260812 00:11:46.111851 3838 heartbeater.cc:347] Connected to a master server at 127.31.227.62:33097
I20260812 00:11:46.111939 3838 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:46.112141 3838 heartbeater.cc:511] Master 127.31.227.62:33097 requested a full tablet report, sending...
I20260812 00:11:46.112627 3194 ts_manager.cc:194] Registered new tserver with Master: ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303)
I20260812 00:11:46.113130 3194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:37911
I20260812 00:11:46.116770 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3573
I20260812 00:11:46.116884 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:46.117070 32652 external_mini_cluster.cc:1428] Setting key b8f43dd0508a6314661c21908c644359
I20260812 00:11:46.119055 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.113066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:53961 (local address 127.31.227.62:33097)
0812 00:11:46.113107 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.113108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.113240 (+ 132us) server_negotiation.cc:407] Connection header received
0812 00:11:46.113315 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.113316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.113326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.113381 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:46.113811 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.114272 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.114991 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.115197 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.115268 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.116226 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:46.116228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:46.116228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:46.116235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:46.117887 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.118230 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.118231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.118232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.118285 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.118495 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.118496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.118496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.118643 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:46.118717 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.118882 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.118960 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:11:46.119067 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.113000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.31.227.2:53961)
0812 00:11:46.113173 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.113182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.113255 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.113387 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.113388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.113644 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:46.113726 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.113728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.114370 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.114371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.114884 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.114885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.115280 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.115281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.115369 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.115816 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:46.115822 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:46.116105 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:46.117964 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.117965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.117966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.118142 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.118359 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.118360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.118361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.118420 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.118723 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:46.118724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:46.118772 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.118856 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.118967 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:46.142621 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33097
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:46.227496 3850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:46.227741 3850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:46.227787 3850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:46.228963 3850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:46.229033 3850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:46.229066 3850 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:11:46.229096 3850 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:11:46.229130 3850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:46.230643 3850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:46.230919 3850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:46.231206 3850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:46.236402 3865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:46.236467 3864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:46.236425 3869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:46.236768 3850 server_base.cc:1034] running on GCE node
I20260812 00:11:46.236994 3850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:46.237308 3850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:46.238461 3850 hybrid_clock.cc:648] HybridClock initialized: now 1786493506238451 us; error 28 us; skew 500 ppm
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493506, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:46.240955 3850 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:46.241403 3850 webserver.cc:460] Webserver started at http://127.31.227.3:45005/ using document root <none> and password file <none>
I20260812 00:11:46.241691 3850 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:46.241765 3850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:46.241930 3850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:46.243034 3850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "639d2464e36d412e8a916d486b299784"
format_stamp: "Formatted at 2026-08-12 00:11:46 on dist-test-slave-zr1t"
server_key: "08747ecdab0694e755cd52f4895b7ef5"
server_key_iv: "887dafecd0691015da315ca0357c2d6d"
server_key_version: "encryptionkey@0"
I20260812 00:11:46.243422 3850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "639d2464e36d412e8a916d486b299784"
format_stamp: "Formatted at 2026-08-12 00:11:46 on dist-test-slave-zr1t"
server_key: "08747ecdab0694e755cd52f4895b7ef5"
server_key_iv: "887dafecd0691015da315ca0357c2d6d"
server_key_version: "encryptionkey@0"
I20260812 00:11:46.244829 3850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:46.245664 3884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:46.245847 3850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:46.246006 3850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "639d2464e36d412e8a916d486b299784"
format_stamp: "Formatted at 2026-08-12 00:11:46 on dist-test-slave-zr1t"
server_key: "08747ecdab0694e755cd52f4895b7ef5"
server_key_iv: "887dafecd0691015da315ca0357c2d6d"
server_key_version: "encryptionkey@0"
I20260812 00:11:46.246100 3850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:46.269804 3850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:46.271764 3850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:46.271942 3850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:46.272500 3850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:46.272565 3850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:46.272629 3850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:46.272658 3850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493506, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:46.280362 3850 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:43101
I20260812 00:11:46.280453 4083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:43101 every 8 connection(s)
I20260812 00:11:46.280766 3850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:11:46.283367 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.273075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:57597 (local address 127.31.227.62:33097)
0812 00:11:46.273121 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.273123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.273350 (+ 227us) server_negotiation.cc:407] Connection header received
0812 00:11:46.273963 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.273964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.273977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.274032 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:46.274938 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.275744 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.277405 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.277746 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.277818 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.280617 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:46.280618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:46.280619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:46.280626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:46.282221 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.282617 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.282618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.282618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.282669 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.282842 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.282843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.282843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.282952 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:46.283002 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.283241 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.283304 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:11:46.283566 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.273063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.31.227.3:57597)
0812 00:11:46.273227 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.273264 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.273851 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.274118 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.274121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.274450 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:46.274830 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.274851 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.275854 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.275856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.277301 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.277305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.277874 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.277875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.278037 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.278545 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:46.278558 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:46.280502 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:46.282328 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.282332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.282334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.282549 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.282710 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.282710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.282710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.282794 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.283011 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:46.283017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:46.283129 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.283212 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.283331 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260812 00:11:46.284070 4085 heartbeater.cc:347] Connected to a master server at 127.31.227.62:33097
I20260812 00:11:46.284157 4085 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:46.284380 4085 heartbeater.cc:511] Master 127.31.227.62:33097 requested a full tablet report, sending...
I20260812 00:11:46.284770 3194 ts_manager.cc:194] Registered new tserver with Master: 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:46.285177 3194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:57597
I20260812 00:11:46.288503 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3850
I20260812 00:11:46.288602 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:46.288821 32652 external_mini_cluster.cc:1428] Setting key 225e54e7812cbecd7fe778dea37154df
I20260812 00:11:46.289633 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:11:46.291144 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.285250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:49743 (local address 127.31.227.62:33097)
0812 00:11:46.285352 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.285353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.285469 (+ 116us) server_negotiation.cc:407] Connection header received
0812 00:11:46.285551 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.285552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.285562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.285618 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:46.285936 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.286403 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.287100 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.287307 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.287382 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.288364 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:46.288365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:46.288366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:46.288372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:46.290002 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.290346 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.290347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.290348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.290401 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.290613 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.290614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.290614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.290723 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:46.290794 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.290971 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.291044 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 00:11:46.291158 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.285161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.31.227.3:49743)
0812 00:11:46.285402 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.285411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.285482 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.285623 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.285624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.285800 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:46.285860 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.285861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.286504 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.286505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.287017 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.287019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.287381 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.287382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.287470 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.287925 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:46.287930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:46.288244 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:46.290084 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.290085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.290086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.290259 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.290478 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.290479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.290479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.290539 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.290798 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:46.290800 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:46.290857 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.290939 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.291046 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:46.314109 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.306973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.31.227.62:33097)
0812 00:11:46.307042 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.307044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.307054 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:46.307089 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.307090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.307099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.307163 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:46.307765 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.308262 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.309017 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.309214 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.309289 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.311641 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:46.311643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:46.311644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:46.311658 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:46.313038 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.313338 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.313339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.313339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.313389 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.313616 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.313617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.313617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.313720 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:46.313795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.313920 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.313990 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:11:46.315757 3194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:46.317417 3194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:46.324546 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.319914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43101 (local address 127.0.0.1:53690)
0812 00:11:46.320072 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.320115 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.320188 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.320561 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.320564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.320566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.320634 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.320635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.322103 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.322104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.323272 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.323275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.324216 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.324217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.324295 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.324308 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.324389 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.324445 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":36,"thread_start_us":115,"threads_started":1}
I20260812 00:11:46.324716 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.320427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.31.227.1:44469)
0812 00:11:46.320596 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.320601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.320612 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:11:46.320659 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.320661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.320678 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.320767 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.321038 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.322166 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.323546 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.324279 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.324338 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.324390 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.324556 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.324599 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260812 00:11:46.324841 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.319914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44469 (local address 127.0.0.1:50944)
0812 00:11:46.320100 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.320130 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.320188 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.320909 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.320910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.320912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.320970 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.320971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.322256 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.322257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.323454 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.323457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.324361 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.324362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.324433 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.324437 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.324523 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.324571 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":111,"threads_started":1}
I20260812 00:11:46.324916 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.320184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.31.227.3:43101)
0812 00:11:46.320384 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.320389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.320399 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:46.320445 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.320447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.320460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.320542 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.320760 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.322012 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.323369 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.324124 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.324205 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.324268 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.324749 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.324796 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260812 00:11:46.324971 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.319917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:39303 (local address 127.0.0.1:51824)
0812 00:11:46.320279 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.320288 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.320326 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.320940 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.320941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.320943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.320972 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.320974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.322602 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.322603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.323671 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.323673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.324498 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.324498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.324578 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.324582 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.324843 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.324892 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20260812 00:11:46.325274 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.320202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.31.227.2:39303)
0812 00:11:46.320672 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.320676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.320686 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:46.320725 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.320727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.320785 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.320872 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.321606 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.322511 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.323758 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.324337 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.324406 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.324462 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.325123 (+ 661us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.325170 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":86,"threads_started":1}
I20260812 00:11:46.325812 3970 tablet_service.cc:1467] Processing CreateTablet for tablet c261187b305d4b63a8c2027f56987ed0 (DEFAULT_TABLE table=test-table [id=436411c5ff2d4894bbb1f54acca68c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:46.325814 3434 tablet_service.cc:1467] Processing CreateTablet for tablet c261187b305d4b63a8c2027f56987ed0 (DEFAULT_TABLE table=test-table [id=436411c5ff2d4894bbb1f54acca68c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:46.326135 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c261187b305d4b63a8c2027f56987ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.326177 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c261187b305d4b63a8c2027f56987ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.326129 3700 tablet_service.cc:1467] Processing CreateTablet for tablet c261187b305d4b63a8c2027f56987ed0 (DEFAULT_TABLE table=test-table [id=436411c5ff2d4894bbb1f54acca68c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:46.326485 3700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c261187b305d4b63a8c2027f56987ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.328902 4130 tablet_bootstrap.cc:492] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Bootstrap starting.
I20260812 00:11:46.329843 4130 tablet_bootstrap.cc:654] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.330173 4134 tablet_bootstrap.cc:492] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Bootstrap starting.
I20260812 00:11:46.330274 4130 log.cc:824] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:46.330658 4131 tablet_bootstrap.cc:492] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Bootstrap starting.
I20260812 00:11:46.330981 4134 tablet_bootstrap.cc:654] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.331089 4130 tablet_bootstrap.cc:492] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: No bootstrap required, opened a new log
I20260812 00:11:46.331167 4130 ts_tablet_manager.cc:1397] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:11:46.331303 4131 tablet_bootstrap.cc:654] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.331324 4134 log.cc:824] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:46.331979 4131 log.cc:824] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:46.332154 4134 tablet_bootstrap.cc:492] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: No bootstrap required, opened a new log
I20260812 00:11:46.332228 4134 ts_tablet_manager.cc:1397] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:11:46.332790 4131 tablet_bootstrap.cc:492] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: No bootstrap required, opened a new log
I20260812 00:11:46.332867 4131 ts_tablet_manager.cc:1397] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:11:46.333391 4130 raft_consensus.cc:348] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.333619 4130 raft_consensus.cc:374] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.333668 4130 raft_consensus.cc:729] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639d2464e36d412e8a916d486b299784, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.333782 4130 consensus_queue.cc:260] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.334098 4085 heartbeater.cc:503] Master 127.31.227.62:33097 was elected leader, sending a full tablet report...
I20260812 00:11:46.334044 4134 raft_consensus.cc:348] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.334205 4134 raft_consensus.cc:374] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.334254 4134 raft_consensus.cc:729] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.334368 4130 ts_tablet_manager.cc:1428] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 00:11:46.334375 4134 consensus_queue.cc:260] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.334659 4134 ts_tablet_manager.cc:1428] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:11:46.335050 4131 raft_consensus.cc:348] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.335227 4131 raft_consensus.cc:374] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.335273 4131 raft_consensus.cc:729] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.335399 4131 consensus_queue.cc:260] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.335698 4131 ts_tablet_manager.cc:1428] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:11:46.335749 3552 heartbeater.cc:503] Master 127.31.227.62:33097 was elected leader, sending a full tablet report...
I20260812 00:11:46.335819 3838 heartbeater.cc:503] Master 127.31.227.62:33097 was elected leader, sending a full tablet report...
I20260812 00:11:46.358141 4140 raft_consensus.cc:482] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:46.358268 4140 raft_consensus.cc:504] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.358645 4140 leader_election.cc:288] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101), ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303)
W20260812 00:11:46.359056 3839 tablet.cc:2369] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:11:46.361720 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.359033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:55467 (local address 127.31.227.3:43101)
0812 00:11:46.359108 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.359110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.359118 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:11:46.359148 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.359149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.359159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.359209 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.359451 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.359933 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.360927 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.361321 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.361409 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.361418 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.361607 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.361650 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:11:46.361722 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.358838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43101 (local address 127.31.227.1:55467)
0812 00:11:46.358902 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.358914 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.358974 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.359263 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.359264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.359266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.359349 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.359351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.360021 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.360022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.360820 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.360822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.361409 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.361410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.361503 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.361506 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.361599 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.361646 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":43}
I20260812 00:11:46.361809 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.358973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:46731 (local address 127.31.227.2:39303)
0812 00:11:46.359064 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.359066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.359219 (+ 153us) server_negotiation.cc:407] Connection header received
0812 00:11:46.359326 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.359327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.359339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.359412 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.359585 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.360050 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.360985 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.361375 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.361464 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.361474 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.361680 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.361722 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 00:11:46.361980 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.358861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:39303 (local address 127.31.227.1:46731)
0812 00:11:46.359146 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.359158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.359247 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.359418 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.359418 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.359420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.359507 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.359508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.360139 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.360141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.360888 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.360890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.361466 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.361467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.361583 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.361587 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.361659 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.361702 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":130,"threads_started":1}
I20260812 00:11:46.362123 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261187b305d4b63a8c2027f56987ed0" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784" is_pre_election: true
I20260812 00:11:46.362324 4003 raft_consensus.cc:2454] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 0.
I20260812 00:11:46.362421 3742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261187b305d4b63a8c2027f56987ed0" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" is_pre_election: true
I20260812 00:11:46.362610 3742 raft_consensus.cc:2454] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 0.
I20260812 00:11:46.362602 3355 leader_election.cc:302] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639d2464e36d412e8a916d486b299784, b6435b61b6cc4f61abc372644ef97b13; no voters:
I20260812 00:11:46.362751 4140 raft_consensus.cc:2781] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:46.362823 4140 raft_consensus.cc:482] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:46.362854 4140 raft_consensus.cc:3037] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.363626 4140 raft_consensus.cc:504] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.363781 4140 leader_election.cc:288] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 election: Requested vote from peers 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101), ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303)
I20260812 00:11:46.363965 3742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261187b305d4b63a8c2027f56987ed0" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0"
I20260812 00:11:46.364027 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261187b305d4b63a8c2027f56987ed0" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784"
I20260812 00:11:46.364065 3742 raft_consensus.cc:3037] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.364099 4003 raft_consensus.cc:3037] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.365007 3742 raft_consensus.cc:2454] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 1.
I20260812 00:11:46.365007 4003 raft_consensus.cc:2454] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 1.
I20260812 00:11:46.365201 3356 leader_election.cc:302] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6435b61b6cc4f61abc372644ef97b13, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:46.365271 4140 raft_consensus.cc:2781] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:46.365438 4140 raft_consensus.cc:686] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 LEADER]: Becoming Leader. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Running, Role: LEADER
I20260812 00:11:46.365578 4140 consensus_queue.cc:237] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } }
I20260812 00:11:46.366508 3194 catalog_manager.cc:5495] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "b6435b61b6cc4f61abc372644ef97b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } health_report { overall_health: HEALTHY } } }
Aug 12 00:11:46 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493505, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:46.401155 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.394341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.31.227.62:33097)
0812 00:11:46.394407 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.394409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.394547 (+ 138us) server_negotiation.cc:407] Connection header received
0812 00:11:46.394610 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.394611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.394622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.394666 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:46.395112 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.395595 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.396255 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.396442 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.396497 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.398594 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:46.398596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:46.398596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:46.398603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:46.400135 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.400477 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.400478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.400479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.400553 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:46.400734 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:46.400735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:46.400735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:46.400852 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:46.400905 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.401018 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.401090 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:11:46.401312 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.394252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33097 (local address 127.0.0.1:59428)
0812 00:11:46.394478 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.394488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.394545 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.394691 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.394692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.394993 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:46.395044 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.395046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.395664 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.395665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.396180 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.396181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.396510 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.396511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.396553 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.397076 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:46.397080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:46.398497 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:46.400205 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.400211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.400213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.400392 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.400623 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:46.400623 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:46.400624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:46.400678 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:46.400907 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:46.400913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:46.400952 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.401001 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.401161 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":71,"thread_start_us":107,"threads_started":1}
I20260812 00:11:46.403410 3194 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:59428:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:11:46.403654 3194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:46.405238 3700 tablet_service.cc:1467] Processing CreateTablet for tablet cf3effeee1144ed9b0a8b6eb42db5df0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=327cfc82df994db1bab596d0683866ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:11:46.405251 3970 tablet_service.cc:1467] Processing CreateTablet for tablet cf3effeee1144ed9b0a8b6eb42db5df0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=327cfc82df994db1bab596d0683866ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:11:46.405435 3700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3effeee1144ed9b0a8b6eb42db5df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.405438 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3effeee1144ed9b0a8b6eb42db5df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.405426 3434 tablet_service.cc:1467] Processing CreateTablet for tablet cf3effeee1144ed9b0a8b6eb42db5df0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=327cfc82df994db1bab596d0683866ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:11:46.405619 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3effeee1144ed9b0a8b6eb42db5df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.406919 4134 tablet_bootstrap.cc:492] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: Bootstrap starting.
I20260812 00:11:46.407133 4131 tablet_bootstrap.cc:492] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: Bootstrap starting.
I20260812 00:11:46.407663 4131 tablet_bootstrap.cc:654] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.407696 4134 tablet_bootstrap.cc:654] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.408094 4130 tablet_bootstrap.cc:492] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: Bootstrap starting.
I20260812 00:11:46.408468 4131 tablet_bootstrap.cc:492] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: No bootstrap required, opened a new log
I20260812 00:11:46.408474 4134 tablet_bootstrap.cc:492] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: No bootstrap required, opened a new log
I20260812 00:11:46.408515 4134 ts_tablet_manager.cc:1397] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:11:46.408515 4131 ts_tablet_manager.cc:1397] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:46.408699 4130 tablet_bootstrap.cc:654] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.408679 4134 raft_consensus.cc:348] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.408679 4131 raft_consensus.cc:348] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.408759 4131 raft_consensus.cc:374] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.408759 4134 raft_consensus.cc:374] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.408783 4134 raft_consensus.cc:729] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.408783 4131 raft_consensus.cc:729] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.408835 4131 consensus_queue.cc:260] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.408835 4134 consensus_queue.cc:260] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.408921 4131 ts_tablet_manager.cc:1428] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:46.408921 4134 ts_tablet_manager.cc:1428] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:46.409060 4144 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.409017 4139 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.409139 4144 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:46.409139 4139 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:46.410018 4130 tablet_bootstrap.cc:492] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: No bootstrap required, opened a new log
I20260812 00:11:46.410090 4130 ts_tablet_manager.cc:1397] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260812 00:11:46.410240 4130 raft_consensus.cc:348] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.410307 4130 raft_consensus.cc:374] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.410321 4130 raft_consensus.cc:729] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639d2464e36d412e8a916d486b299784, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.410403 4130 consensus_queue.cc:260] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.410502 4130 ts_tablet_manager.cc:1428] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:46.410568 4138 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.410645 4138 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 00:11:46.440917 3556 tablet.cc:2369] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:11:46.443845 4138 raft_consensus.cc:482] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:46.443933 4138 raft_consensus.cc:504] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.444442 4138 leader_election.cc:288] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469), ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303)
I20260812 00:11:46.447603 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.444612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:39303 (local address 127.31.227.3:60853)
0812 00:11:46.444681 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.444696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.444749 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.444952 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.444954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.444956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.445050 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.445052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.445890 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.445891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.446766 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.446769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.447321 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.447322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.447413 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.447416 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.447485 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.447530 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260812 00:11:46.447603 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.444755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:60853 (local address 127.31.227.2:39303)
0812 00:11:46.444823 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.444825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.444833 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:11:46.444860 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.444861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.444878 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.444959 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.445129 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.445806 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.446864 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.447236 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.447322 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.447331 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.447491 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.447532 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:11:46.447737 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.444749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:56429 (local address 127.31.227.1:44469)
0812 00:11:46.444849 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.444851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.444991 (+ 140us) server_negotiation.cc:407] Connection header received
0812 00:11:46.445066 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.445067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.445077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.445149 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.445325 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.445929 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.446971 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.447353 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.447443 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.447452 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.447624 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.447667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 00:11:46.447757 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.444622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44469 (local address 127.31.227.3:56429)
0812 00:11:46.444909 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.444919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.445008 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.445154 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.445155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.445157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.445246 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.445248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.446024 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.446025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.446872 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.446875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.447465 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.447467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.447544 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.447548 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.447618 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.447660 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":72,"thread_start_us":109,"threads_started":1}
I20260812 00:11:46.448027 3742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3effeee1144ed9b0a8b6eb42db5df0" candidate_uuid: "639d2464e36d412e8a916d486b299784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" is_pre_election: true
I20260812 00:11:46.448146 3742 raft_consensus.cc:2454] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639d2464e36d412e8a916d486b299784 in term 0.
I20260812 00:11:46.448094 3465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3effeee1144ed9b0a8b6eb42db5df0" candidate_uuid: "639d2464e36d412e8a916d486b299784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6435b61b6cc4f61abc372644ef97b13" is_pre_election: true
I20260812 00:11:46.448215 3465 raft_consensus.cc:2454] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639d2464e36d412e8a916d486b299784 in term 0.
I20260812 00:11:46.448334 3891 leader_election.cc:302] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639d2464e36d412e8a916d486b299784, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:46.448488 4138 raft_consensus.cc:2781] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:46.448541 4138 raft_consensus.cc:482] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:46.448570 4138 raft_consensus.cc:3037] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.449369 4138 raft_consensus.cc:504] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.449523 4138 leader_election.cc:288] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [CANDIDATE]: Term 1 election: Requested vote from peers b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469), ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303)
I20260812 00:11:46.449687 3465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3effeee1144ed9b0a8b6eb42db5df0" candidate_uuid: "639d2464e36d412e8a916d486b299784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6435b61b6cc4f61abc372644ef97b13"
I20260812 00:11:46.449761 3465 raft_consensus.cc:3037] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.449753 3742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3effeee1144ed9b0a8b6eb42db5df0" candidate_uuid: "639d2464e36d412e8a916d486b299784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0"
I20260812 00:11:46.449848 3742 raft_consensus.cc:3037] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.450491 3465 raft_consensus.cc:2454] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639d2464e36d412e8a916d486b299784 in term 1.
I20260812 00:11:46.450621 3742 raft_consensus.cc:2454] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639d2464e36d412e8a916d486b299784 in term 1.
I20260812 00:11:46.450697 3888 leader_election.cc:302] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639d2464e36d412e8a916d486b299784, b6435b61b6cc4f61abc372644ef97b13; no voters:
I20260812 00:11:46.450805 4138 raft_consensus.cc:2781] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:46.450965 4138 raft_consensus.cc:686] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [term 1 LEADER]: Becoming Leader. State: Replica: 639d2464e36d412e8a916d486b299784, State: Running, Role: LEADER
I20260812 00:11:46.451079 4138 consensus_queue.cc:237] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.451552 4171 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: TxnStatusTablet state changed. Reason: New leader 639d2464e36d412e8a916d486b299784. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.451648 4171 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:11:46.451911 4173 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:11:46.452052 4173 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:11:46.451970 3194 catalog_manager.cc:5495] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 reported cstate change: term changed from 0 to 1, leader changed from <none> to 639d2464e36d412e8a916d486b299784 (127.31.227.3). New cstate: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } health_report { overall_health: HEALTHY } } }
I20260812 00:11:46.459828 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.456965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43101 (local address 127.0.0.1:53702)
0812 00:11:46.457058 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.457075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.457123 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.457292 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.457293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.457294 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:46.457342 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.457344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.458316 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.458317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.458873 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.458875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.459351 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.459352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.459394 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.459410 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.459695 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.459701 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.459730 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.459771 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 00:11:46.459842 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.457023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.31.227.3:43101)
0812 00:11:46.457075 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.457077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.457108 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:11:46.457186 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.457187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.457207 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.457283 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.457405 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.458228 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.458959 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.459265 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.459346 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.459499 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.459627 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.459715 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.459737 (+ 22us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.459763 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:11:46.462152 3465 raft_consensus.cc:1261] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Refusing update from remote peer 639d2464e36d412e8a916d486b299784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:46.462190 3742 raft_consensus.cc:1261] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Refusing update from remote peer 639d2464e36d412e8a916d486b299784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:46.462558 4138 consensus_queue.cc:1035] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:46.462716 4171 consensus_queue.cc:1035] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:46.463902 4144 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: TxnStatusTablet state changed. Reason: New leader 639d2464e36d412e8a916d486b299784. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.464006 4144 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:46.464846 4139 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: TxnStatusTablet state changed. Reason: New leader 639d2464e36d412e8a916d486b299784. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.464951 4139 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:46.465602 4171 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.465657 4139 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.465720 4171 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:11:46.465761 4139 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P ff533b3951cc4dc9b6cc22e05c32daa0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:46.466105 4182 mvcc.cc:205] Tried to move back new op lower bound from 7317477402465984512 to 7317477402424385536. Current Snapshot: MvccSnapshot[applied={T|T < 7317477402465984512}]
I20260812 00:11:46.466326 4138 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.466413 4138 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P 639d2464e36d412e8a916d486b299784: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:11:46.466647 4144 tablet_replica.cc:439] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "639d2464e36d412e8a916d486b299784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } } }
I20260812 00:11:46.466727 4144 tablet_replica.cc:442] T cf3effeee1144ed9b0a8b6eb42db5df0 P b6435b61b6cc4f61abc372644ef97b13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:11:46.467005 4181 mvcc.cc:205] Tried to move back new op lower bound from 7317477402465984512 to 7317477402424385536. Current Snapshot: MvccSnapshot[applied={T|T < 7317477402465984512}]
I20260812 00:11:46.473938 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.471505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.31.227.1:44469)
0812 00:11:46.471562 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.471564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.471571 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:11:46.471611 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.471611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.471626 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.471704 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.471814 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.472328 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.473142 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.473351 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.473426 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.473617 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.473723 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.473809 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.473836 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.473860 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:11:46.479382 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.476572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43101 (local address 127.31.227.1:36403)
0812 00:11:46.476736 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.476747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.476807 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.476930 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.476931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.476932 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:46.476975 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.476976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.477861 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.477862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.478422 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.478423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.478916 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.478916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.478996 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.479033 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.479205 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.479207 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.479288 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.479326 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260812 00:11:46.479401 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.476655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:36403 (local address 127.31.227.3:43101)
0812 00:11:46.476700 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.476702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.476816 (+ 114us) server_negotiation.cc:407] Connection header received
0812 00:11:46.476847 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.476848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.476860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.476931 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.477012 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.477786 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.478513 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.478845 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.478916 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.479107 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.479143 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.479198 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.479297 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.479317 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:11:46.482203 3742 raft_consensus.cc:1261] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Refusing update from remote peer b6435b61b6cc4f61abc372644ef97b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:46.482240 4003 raft_consensus.cc:1261] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Refusing update from remote peer b6435b61b6cc4f61abc372644ef97b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:46.482566 4140 consensus_queue.cc:1035] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:46.482695 4144 consensus_queue.cc:1035] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:46.484817 4180 mvcc.cc:205] Tried to move back new op lower bound from 7317477402549153792 to 7317477402074030080. Current Snapshot: MvccSnapshot[applied={T|T < 7317477402549153792}]
I20260812 00:11:46.492300 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.489457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44469 (local address 127.31.227.3:32921)
0812 00:11:46.489572 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.489584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.489637 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.489766 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.489766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.489767 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:46.489853 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.489860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.490509 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.490511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.491424 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.491445 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.491908 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.491909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.491969 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.491984 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.492169 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.492171 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.492206 (+ 35us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.492245 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:11:46.492309 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.489547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:32921 (local address 127.31.227.1:44469)
0812 00:11:46.489584 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.489586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.489620 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:11:46.489695 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.489696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.489707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.489762 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.489934 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.490407 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.491525 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.491826 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.491904 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.492065 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.492105 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.492159 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.492210 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.492233 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:11:46.501572 3194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:46.501689 3194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:46.502554 3194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Sending DeleteTablet for 3 replicas of tablet c261187b305d4b63a8c2027f56987ed0
I20260812 00:11:46.502921 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet c261187b305d4b63a8c2027f56987ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:46 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 00:11:46.502925 3700 tablet_service.cc:1514] Processing DeleteTablet for tablet c261187b305d4b63a8c2027f56987ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:46 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:51824
I20260812 00:11:46.503171 4219 tablet_replica.cc:330] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: stopping tablet replica
I20260812 00:11:46.503284 4219 raft_consensus.cc:2229] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:46.503314 3970 tablet_service.cc:1514] Processing DeleteTablet for tablet c261187b305d4b63a8c2027f56987ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:46 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:53690
I20260812 00:11:46.503504 4219 raft_consensus.cc:2258] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:46.503768 3194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:46.503930 3194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:46.504026 4219 ts_tablet_manager.cc:1905] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:46.505077 3700 tablet_service.cc:1467] Processing CreateTablet for tablet 0300f1db699c49f0a7314453fa586ace (DEFAULT_TABLE table=test-table [id=ed749dbbcdee4ecbbc32dafc7e406f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:46.505218 3700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0300f1db699c49f0a7314453fa586ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.505818 4219 ts_tablet_manager.cc:1918] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:46.505904 4219 log.cc:1196] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/c261187b305d4b63a8c2027f56987ed0
I20260812 00:11:46.506175 4219 ts_tablet_manager.cc:1939] T c261187b305d4b63a8c2027f56987ed0 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting consensus metadata
I20260812 00:11:46.506636 3172 catalog_manager.cc:4841] TS ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303): tablet c261187b305d4b63a8c2027f56987ed0 (table test-table [id=436411c5ff2d4894bbb1f54acca68c5a]) successfully deleted
I20260812 00:11:46.506668 4134 tablet_bootstrap.cc:492] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Bootstrap starting.
I20260812 00:11:46.507095 4134 tablet_bootstrap.cc:654] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.507633 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 0300f1db699c49f0a7314453fa586ace (DEFAULT_TABLE table=test-table [id=ed749dbbcdee4ecbbc32dafc7e406f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:46.507769 4134 tablet_bootstrap.cc:492] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: No bootstrap required, opened a new log
I20260812 00:11:46.507825 4134 ts_tablet_manager.cc:1397] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:46.507973 4134 raft_consensus.cc:348] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.508050 4134 raft_consensus.cc:374] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.508080 4134 raft_consensus.cc:729] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.508119 4134 consensus_queue.cc:260] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.508214 4134 ts_tablet_manager.cc:1428] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:46.508271 3970 tablet_service.cc:1467] Processing CreateTablet for tablet 0300f1db699c49f0a7314453fa586ace (DEFAULT_TABLE table=test-table [id=ed749dbbcdee4ecbbc32dafc7e406f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:46.508484 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0300f1db699c49f0a7314453fa586ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.509615 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0300f1db699c49f0a7314453fa586ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:46.509912 4130 tablet_bootstrap.cc:492] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Bootstrap starting.
I20260812 00:11:46.510419 4130 tablet_bootstrap.cc:654] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.511390 4131 tablet_bootstrap.cc:492] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Bootstrap starting.
I20260812 00:11:46.511551 4130 tablet_bootstrap.cc:492] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: No bootstrap required, opened a new log
I20260812 00:11:46.511629 4130 ts_tablet_manager.cc:1397] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 00:11:46.511786 4224 tablet_replica.cc:330] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: stopping tablet replica
I20260812 00:11:46.511854 4131 tablet_bootstrap.cc:654] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:46.511919 4224 raft_consensus.cc:2229] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:46.511976 4224 raft_consensus.cc:2258] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:46.511878 4130 raft_consensus.cc:348] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.512050 4130 raft_consensus.cc:374] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.512130 4130 raft_consensus.cc:729] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639d2464e36d412e8a916d486b299784, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.512176 4223 tablet_replica.cc:330] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: stopping tablet replica
I20260812 00:11:46.512219 4130 consensus_queue.cc:260] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.512312 4223 raft_consensus.cc:2229] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:11:46.512490 4223 raft_consensus.cc:2258] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:46.512429 4130 ts_tablet_manager.cc:1428] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 00:11:46.513197 4223 ts_tablet_manager.cc:1905] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:46.514906 4223 ts_tablet_manager.cc:1918] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:46.514997 4223 log.cc:1196] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/c261187b305d4b63a8c2027f56987ed0
I20260812 00:11:46.515278 4223 ts_tablet_manager.cc:1939] T c261187b305d4b63a8c2027f56987ed0 P b6435b61b6cc4f61abc372644ef97b13: Deleting consensus metadata
I20260812 00:11:46.515705 3169 catalog_manager.cc:4841] TS b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469): tablet c261187b305d4b63a8c2027f56987ed0 (table test-table [id=436411c5ff2d4894bbb1f54acca68c5a]) successfully deleted
I20260812 00:11:46.515966 4131 tablet_bootstrap.cc:492] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: No bootstrap required, opened a new log
I20260812 00:11:46.516038 4131 ts_tablet_manager.cc:1397] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 00:11:46.516192 4131 raft_consensus.cc:348] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.516269 4131 raft_consensus.cc:374] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:46.516302 4131 raft_consensus.cc:729] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Initialized, Role: FOLLOWER
I20260812 00:11:46.516351 4131 consensus_queue.cc:260] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.516448 4131 ts_tablet_manager.cc:1428] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:46.512485 4224 ts_tablet_manager.cc:1905] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:46.523044 4224 ts_tablet_manager.cc:1918] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:46.523131 4224 log.cc:1196] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/c261187b305d4b63a8c2027f56987ed0
I20260812 00:11:46.523397 4224 ts_tablet_manager.cc:1939] T c261187b305d4b63a8c2027f56987ed0 P 639d2464e36d412e8a916d486b299784: Deleting consensus metadata
I20260812 00:11:46.523859 3170 catalog_manager.cc:4841] TS 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101): tablet c261187b305d4b63a8c2027f56987ed0 (table test-table [id=436411c5ff2d4894bbb1f54acca68c5a]) successfully deleted
W20260812 00:11:46.531358 4086 tablet.cc:2369] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:11:46.621798 4139 raft_consensus.cc:482] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:46.621905 4139 raft_consensus.cc:504] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.622288 4139 leader_election.cc:288] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469), 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:46.625584 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.622488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44469 (local address 127.31.227.2:60107)
0812 00:11:46.622705 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.622716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.622775 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.622927 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.622928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.622930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.622996 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.622998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.623687 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.623689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.624531 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.624534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.625282 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.625283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.625363 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.625368 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.625429 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.625473 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":60,"threads_started":1}
I20260812 00:11:46.625589 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.622551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60107 (local address 127.31.227.1:44469)
0812 00:11:46.622605 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.622607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.622790 (+ 183us) server_negotiation.cc:407] Connection header received
0812 00:11:46.622832 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.622833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.622844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.622910 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.623091 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.623600 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.624628 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.625177 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.625256 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.625265 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.625447 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.625484 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:11:46.625932 3465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300f1db699c49f0a7314453fa586ace" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6435b61b6cc4f61abc372644ef97b13" is_pre_election: true
I20260812 00:11:46.626030 3465 raft_consensus.cc:2454] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 0.
I20260812 00:11:46.626237 3606 leader_election.cc:302] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6435b61b6cc4f61abc372644ef97b13, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:46.626397 4139 raft_consensus.cc:2781] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:46.626454 4139 raft_consensus.cc:482] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:46.626478 4139 raft_consensus.cc:3037] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.626840 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.622476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43101 (local address 127.31.227.2:35373)
0812 00:11:46.622613 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.622625 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.622722 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.623712 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.623714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.623717 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:46.623778 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.623779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.624664 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.624665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.625845 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.625848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.626588 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.626589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.626671 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.626675 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.626730 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.626768 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260812 00:11:46.626869 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.622651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:35373 (local address 127.31.227.3:43101)
0812 00:11:46.622703 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.622705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.622717 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:11:46.622979 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.622980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.622989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.623049 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:46.623866 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.624572 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.625953 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.626468 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.626519 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.626528 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.626759 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.626801 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:11:46.627177 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300f1db699c49f0a7314453fa586ace" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784" is_pre_election: true
I20260812 00:11:46.627279 4003 raft_consensus.cc:2454] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 0.
I20260812 00:11:46.627522 4139 raft_consensus.cc:504] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.627676 4139 leader_election.cc:288] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 election: Requested vote from peers b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469), 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:46.628031 3465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300f1db699c49f0a7314453fa586ace" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6435b61b6cc4f61abc372644ef97b13"
I20260812 00:11:46.628062 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300f1db699c49f0a7314453fa586ace" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784"
I20260812 00:11:46.628113 3465 raft_consensus.cc:3037] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.628126 4003 raft_consensus.cc:3037] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:46.628877 3465 raft_consensus.cc:2454] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 1.
I20260812 00:11:46.628877 4003 raft_consensus.cc:2454] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 1.
I20260812 00:11:46.629081 3608 leader_election.cc:302] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639d2464e36d412e8a916d486b299784, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:46.629194 4139 raft_consensus.cc:2781] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:46.629311 4139 raft_consensus.cc:686] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 LEADER]: Becoming Leader. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Running, Role: LEADER
I20260812 00:11:46.629402 4139 consensus_queue.cc:237] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:46.630138 3194 catalog_manager.cc:5495] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2). New cstate: current_term: 1 leader_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } health_report { overall_health: UNKNOWN } } }
I20260812 00:11:46.663750 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.661169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.31.227.2:39303)
0812 00:11:46.661207 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.661209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.661219 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:46.661310 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.661311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.661334 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.661425 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.661602 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.662090 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.662882 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.663094 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.663169 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.663348 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.663469 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.663549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.663610 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.663631 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:11:46.668766 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.666300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:44237 (local address 127.31.227.3:43101)
0812 00:11:46.666341 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.666343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.666547 (+ 204us) server_negotiation.cc:407] Connection header received
0812 00:11:46.666619 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.666620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.666630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.666676 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.666815 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.667275 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.668043 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.668243 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.668314 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.668511 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.668552 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.668614 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.668671 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.668690 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:11:46.669003 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.666220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43101 (local address 127.31.227.2:44237)
0812 00:11:46.666442 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.666452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.666552 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.666678 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.666678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.666680 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:46.666739 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.666740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.667357 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.667359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.667950 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.667951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.668327 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.668328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.668409 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.668423 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.668618 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.668621 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.668668 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.668713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260812 00:11:46.671288 3465 raft_consensus.cc:1261] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Refusing update from remote peer ff533b3951cc4dc9b6cc22e05c32daa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:46.671352 4003 raft_consensus.cc:1261] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Refusing update from remote peer ff533b3951cc4dc9b6cc22e05c32daa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:46.671597 4139 consensus_queue.cc:1035] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:46.671789 4236 consensus_queue.cc:1035] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:46.680428 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.677141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:39303 (local address 127.31.227.3:45221)
0812 00:11:46.677361 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:46.677407 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:46.677457 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:46.677677 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:46.677678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:46.677679 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:11:46.677748 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.677750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.678634 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.678635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.679542 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:46.679545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.680006 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.680007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:46.680065 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.680070 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.680251 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.680253 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:11:46.680293 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 00:11:46.680346 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260812 00:11:46.680884 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:46.677175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:45221 (local address 127.31.227.2:39303)
0812 00:11:46.677225 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:46.677227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:46.677443 (+ 216us) server_negotiation.cc:407] Connection header received
0812 00:11:46.677554 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:46.677555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:46.677565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:46.677609 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:46.677829 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.678542 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.679664 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:46.679902 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:46.679971 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:46.680147 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:46.680182 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:46.680226 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:46.680795 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 00:11:46.680816 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 00:11:46.681156 4182 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 00:11:48.203328 3194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:48.203462 3194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:48.204360 3194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Sending DeleteTablet for 3 replicas of tablet 0300f1db699c49f0a7314453fa586ace
I20260812 00:11:48.204609 3700 tablet_service.cc:1514] Processing DeleteTablet for tablet 0300f1db699c49f0a7314453fa586ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:48 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:51824
I20260812 00:11:48.204648 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet 0300f1db699c49f0a7314453fa586ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:48 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 00:11:48.205204 3970 tablet_service.cc:1514] Processing DeleteTablet for tablet 0300f1db699c49f0a7314453fa586ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:48 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:53690
I20260812 00:11:48.205297 4294 tablet_replica.cc:330] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: stopping tablet replica
I20260812 00:11:48.205265 4295 tablet_replica.cc:330] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: stopping tablet replica
I20260812 00:11:48.205384 4294 raft_consensus.cc:2229] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:11:48.205433 4295 raft_consensus.cc:2229] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:48.205480 4294 raft_consensus.cc:2258] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:48.205483 4295 raft_consensus.cc:2258] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:48.205757 4295 ts_tablet_manager.cc:1905] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:48.205780 4294 ts_tablet_manager.cc:1905] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:48.207513 4295 ts_tablet_manager.cc:1918] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:11:48.207589 4295 log.cc:1196] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/0300f1db699c49f0a7314453fa586ace
I20260812 00:11:48.207823 4295 ts_tablet_manager.cc:1939] T 0300f1db699c49f0a7314453fa586ace P b6435b61b6cc4f61abc372644ef97b13: Deleting consensus metadata
I20260812 00:11:48.207981 4294 ts_tablet_manager.cc:1918] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:11:48.208047 4294 log.cc:1196] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/0300f1db699c49f0a7314453fa586ace
I20260812 00:11:48.208171 3169 catalog_manager.cc:4841] TS b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469): tablet 0300f1db699c49f0a7314453fa586ace (table test-table [id=ed749dbbcdee4ecbbc32dafc7e406f06]) successfully deleted
I20260812 00:11:48.208227 4297 tablet_replica.cc:330] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: stopping tablet replica
I20260812 00:11:48.208309 4297 raft_consensus.cc:2229] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:48.208338 4294 ts_tablet_manager.cc:1939] T 0300f1db699c49f0a7314453fa586ace P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting consensus metadata
I20260812 00:11:48.208377 4297 raft_consensus.cc:2258] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:48.208644 3172 catalog_manager.cc:4841] TS ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303): tablet 0300f1db699c49f0a7314453fa586ace (table test-table [id=ed749dbbcdee4ecbbc32dafc7e406f06]) successfully deleted
I20260812 00:11:48.209879 4297 ts_tablet_manager.cc:1905] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:48.211736 4297 ts_tablet_manager.cc:1918] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:11:48.211809 4297 log.cc:1196] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/0300f1db699c49f0a7314453fa586ace
I20260812 00:11:48.212044 4297 ts_tablet_manager.cc:1939] T 0300f1db699c49f0a7314453fa586ace P 639d2464e36d412e8a916d486b299784: Deleting consensus metadata
I20260812 00:11:48.212448 3170 catalog_manager.cc:4841] TS 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101): tablet 0300f1db699c49f0a7314453fa586ace (table test-table [id=ed749dbbcdee4ecbbc32dafc7e406f06]) successfully deleted
Aug 12 00:11:48 dist-test-slave-zr1t krb5kdc[3121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:11:48 dist-test-slave-zr1t krb5kdc[3121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:48.231359 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:48.223372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.31.227.62:33097)
0812 00:11:48.223542 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:48.223544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:48.223611 (+ 67us) server_negotiation.cc:407] Connection header received
0812 00:11:48.223715 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:48.223716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:48.223728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:48.223840 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:48.224446 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:48.224973 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:48.225853 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:48.226064 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:48.226151 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:48.228605 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:48.228607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:48.228607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:48.228615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:48.230081 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:48.230469 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:48.230470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:48.230471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:48.230532 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:48.230745 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:48.230746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:48.230747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:48.230880 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:48.230990 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:48.231136 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 00:11:48.231231 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260812 00:11:48.232935 3194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:48.233158 3194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:48.234752 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 (DEFAULT_TABLE table=test-table [id=df334539ed5e4665aea9ac4d4b93a6a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:48.234760 3700 tablet_service.cc:1467] Processing CreateTablet for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 (DEFAULT_TABLE table=test-table [id=df334539ed5e4665aea9ac4d4b93a6a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:48.234777 3970 tablet_service.cc:1467] Processing CreateTablet for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 (DEFAULT_TABLE table=test-table [id=df334539ed5e4665aea9ac4d4b93a6a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:48.234915 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:48.234926 3700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:48.234930 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:48.236091 4315 tablet_bootstrap.cc:492] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Bootstrap starting.
I20260812 00:11:48.236387 4316 tablet_bootstrap.cc:492] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Bootstrap starting.
I20260812 00:11:48.236579 4315 tablet_bootstrap.cc:654] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:48.236814 4318 tablet_bootstrap.cc:492] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Bootstrap starting.
I20260812 00:11:48.236896 4316 tablet_bootstrap.cc:654] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:48.237285 4318 tablet_bootstrap.cc:654] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:48.237317 4315 tablet_bootstrap.cc:492] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: No bootstrap required, opened a new log
I20260812 00:11:48.237367 4315 ts_tablet_manager.cc:1397] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:48.237581 4316 tablet_bootstrap.cc:492] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: No bootstrap required, opened a new log
I20260812 00:11:48.237547 4315 raft_consensus.cc:348] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.237638 4315 raft_consensus.cc:374] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:48.237639 4316 ts_tablet_manager.cc:1397] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:48.237672 4315 raft_consensus.cc:729] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Initialized, Role: FOLLOWER
I20260812 00:11:48.237725 4315 consensus_queue.cc:260] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.237753 4316 raft_consensus.cc:348] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.237824 4316 raft_consensus.cc:374] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:48.237850 4316 raft_consensus.cc:729] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Initialized, Role: FOLLOWER
I20260812 00:11:48.237923 4315 ts_tablet_manager.cc:1428] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:48.237957 4316 consensus_queue.cc:260] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.238003 4318 tablet_bootstrap.cc:492] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: No bootstrap required, opened a new log
I20260812 00:11:48.238047 4318 ts_tablet_manager.cc:1397] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:48.238144 4316 ts_tablet_manager.cc:1428] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:48.238188 4318 raft_consensus.cc:348] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.238276 4318 raft_consensus.cc:374] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:48.238302 4318 raft_consensus.cc:729] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639d2464e36d412e8a916d486b299784, State: Initialized, Role: FOLLOWER
I20260812 00:11:48.238348 4318 consensus_queue.cc:260] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.238466 4318 ts_tablet_manager.cc:1428] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:48.256242 4278 raft_consensus.cc:482] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:48.256356 4278 raft_consensus.cc:504] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.256572 4278 leader_election.cc:288] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469), 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:48.256847 3465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fdd1b15d2b42c7a8e107a79d0f8e32" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6435b61b6cc4f61abc372644ef97b13" is_pre_election: true
I20260812 00:11:48.256847 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fdd1b15d2b42c7a8e107a79d0f8e32" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784" is_pre_election: true
I20260812 00:11:48.256964 4003 raft_consensus.cc:2454] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 0.
I20260812 00:11:48.256964 3465 raft_consensus.cc:2454] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 0.
I20260812 00:11:48.257184 3606 leader_election.cc:302] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6435b61b6cc4f61abc372644ef97b13, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:48.257287 4278 raft_consensus.cc:2781] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:48.257347 4278 raft_consensus.cc:482] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:48.257370 4278 raft_consensus.cc:3037] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:48.258210 4278 raft_consensus.cc:504] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.258368 4278 leader_election.cc:288] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 election: Requested vote from peers b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469), 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:48.258551 3465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fdd1b15d2b42c7a8e107a79d0f8e32" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6435b61b6cc4f61abc372644ef97b13"
I20260812 00:11:48.258535 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fdd1b15d2b42c7a8e107a79d0f8e32" candidate_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784"
I20260812 00:11:48.258646 3465 raft_consensus.cc:3037] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:48.258653 4003 raft_consensus.cc:3037] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:48.259311 4003 raft_consensus.cc:2454] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 1.
I20260812 00:11:48.259336 3465 raft_consensus.cc:2454] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff533b3951cc4dc9b6cc22e05c32daa0 in term 1.
I20260812 00:11:48.259507 3608 leader_election.cc:302] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639d2464e36d412e8a916d486b299784, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:48.259621 4278 raft_consensus.cc:2781] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:48.259689 4278 raft_consensus.cc:686] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 LEADER]: Becoming Leader. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Running, Role: LEADER
I20260812 00:11:48.259759 4278 consensus_queue.cc:237] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.260456 3194 catalog_manager.cc:5495] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2). New cstate: current_term: 1 leader_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } health_report { overall_health: UNKNOWN } } }
I20260812 00:11:48.270133 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:48.267627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.31.227.2:39303)
0812 00:11:48.267782 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:48.267784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:48.267790 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:11:48.267810 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:48.267810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:48.267821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:48.267900 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:48.268078 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:48.268578 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:48.269362 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:48.269653 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:48.269722 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:48.269901 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:48.269932 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:48.269972 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:48.270004 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 00:11:48.270025 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260812 00:11:48.272630 4003 raft_consensus.cc:1261] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Refusing update from remote peer ff533b3951cc4dc9b6cc22e05c32daa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:48.272645 3465 raft_consensus.cc:1261] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Refusing update from remote peer ff533b3951cc4dc9b6cc22e05c32daa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:48.272878 4274 consensus_queue.cc:1035] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:48.273005 4278 consensus_queue.cc:1035] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:48.284575 3194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:48.284686 3194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:48.285440 3194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Sending DeleteTablet for 3 replicas of tablet 28fdd1b15d2b42c7a8e107a79d0f8e32
I20260812 00:11:48.285684 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:48 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 00:11:48.285707 3700 tablet_service.cc:1514] Processing DeleteTablet for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:48 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:51824
I20260812 00:11:48.285710 3970 tablet_service.cc:1514] Processing DeleteTablet for tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:48 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:53690
I20260812 00:11:48.285792 4295 tablet_replica.cc:330] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: stopping tablet replica
I20260812 00:11:48.285794 4294 tablet_replica.cc:330] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: stopping tablet replica
I20260812 00:11:48.285808 4297 tablet_replica.cc:330] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: stopping tablet replica
I20260812 00:11:48.285844 4294 raft_consensus.cc:2229] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:11:48.285851 4295 raft_consensus.cc:2229] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:48.285857 4297 raft_consensus.cc:2229] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:48.285892 4295 raft_consensus.cc:2258] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:48.285905 4297 raft_consensus.cc:2258] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:48.285921 4294 raft_consensus.cc:2258] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:48.286295 4297 ts_tablet_manager.cc:1905] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:48.286302 4294 ts_tablet_manager.cc:1905] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:48.286343 3194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:48.286499 3194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:48.287704 3700 tablet_service.cc:1467] Processing CreateTablet for tablet 13d21570e0214f7f9605d73b80c58630 (DEFAULT_TABLE table=test-table [id=f2dc629cd40043c5a7a1a896330e5f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:48.287745 3970 tablet_service.cc:1467] Processing CreateTablet for tablet 13d21570e0214f7f9605d73b80c58630 (DEFAULT_TABLE table=test-table [id=f2dc629cd40043c5a7a1a896330e5f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:48.287813 3700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d21570e0214f7f9605d73b80c58630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:48.287853 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d21570e0214f7f9605d73b80c58630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:48.288158 4294 ts_tablet_manager.cc:1918] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:48.288183 4297 ts_tablet_manager.cc:1918] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:48.288208 4294 log.cc:1196] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/28fdd1b15d2b42c7a8e107a79d0f8e32
I20260812 00:11:48.288232 4297 log.cc:1196] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/28fdd1b15d2b42c7a8e107a79d0f8e32
I20260812 00:11:48.288439 4294 ts_tablet_manager.cc:1939] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting consensus metadata
I20260812 00:11:48.288478 4297 ts_tablet_manager.cc:1939] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P 639d2464e36d412e8a916d486b299784: Deleting consensus metadata
I20260812 00:11:48.288801 3172 catalog_manager.cc:4841] TS ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303): tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 (table test-table [id=df334539ed5e4665aea9ac4d4b93a6a7]) successfully deleted
I20260812 00:11:48.289026 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 13d21570e0214f7f9605d73b80c58630 (DEFAULT_TABLE table=test-table [id=f2dc629cd40043c5a7a1a896330e5f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:48.289103 4295 ts_tablet_manager.cc:1905] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:48.289120 4316 tablet_bootstrap.cc:492] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Bootstrap starting.
I20260812 00:11:48.289145 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d21570e0214f7f9605d73b80c58630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:48.289270 4318 tablet_bootstrap.cc:492] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Bootstrap starting.
I20260812 00:11:48.289489 4316 tablet_bootstrap.cc:654] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:48.289628 4318 tablet_bootstrap.cc:654] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:48.290485 4315 tablet_bootstrap.cc:492] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13: Bootstrap starting.
I20260812 00:11:48.290727 3170 catalog_manager.cc:4841] TS 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101): tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 (table test-table [id=df334539ed5e4665aea9ac4d4b93a6a7]) successfully deleted
I20260812 00:11:48.290805 4316 tablet_bootstrap.cc:492] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: No bootstrap required, opened a new log
I20260812 00:11:48.290850 4316 ts_tablet_manager.cc:1397] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:11:48.290869 4315 tablet_bootstrap.cc:654] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:48.290961 4316 raft_consensus.cc:348] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.291040 4316 raft_consensus.cc:374] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:48.291075 4316 raft_consensus.cc:729] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff533b3951cc4dc9b6cc22e05c32daa0, State: Initialized, Role: FOLLOWER
I20260812 00:11:48.291061 4295 ts_tablet_manager.cc:1918] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:11:48.291112 4295 log.cc:1196] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/28fdd1b15d2b42c7a8e107a79d0f8e32
I20260812 00:11:48.291126 4316 consensus_queue.cc:260] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.291239 4316 ts_tablet_manager.cc:1428] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:48.291278 4318 tablet_bootstrap.cc:492] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: No bootstrap required, opened a new log
I20260812 00:11:48.291329 4318 ts_tablet_manager.cc:1397] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 00:11:48.291349 4295 ts_tablet_manager.cc:1939] T 28fdd1b15d2b42c7a8e107a79d0f8e32 P b6435b61b6cc4f61abc372644ef97b13: Deleting consensus metadata
I20260812 00:11:48.291430 4315 tablet_bootstrap.cc:492] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13: No bootstrap required, opened a new log
I20260812 00:11:48.291474 4315 ts_tablet_manager.cc:1397] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:48.291460 4318 raft_consensus.cc:348] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.291529 4318 raft_consensus.cc:374] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:48.291548 4318 raft_consensus.cc:729] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639d2464e36d412e8a916d486b299784, State: Initialized, Role: FOLLOWER
I20260812 00:11:48.291590 4318 consensus_queue.cc:260] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.291576 4315 raft_consensus.cc:348] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.291640 4315 raft_consensus.cc:374] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:48.291662 4315 raft_consensus.cc:729] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Initialized, Role: FOLLOWER
I20260812 00:11:48.291685 4318 ts_tablet_manager.cc:1428] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:48.291702 4315 consensus_queue.cc:260] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.291795 4315 ts_tablet_manager.cc:1428] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:48.291931 3169 catalog_manager.cc:4841] TS b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469): tablet 28fdd1b15d2b42c7a8e107a79d0f8e32 (table test-table [id=df334539ed5e4665aea9ac4d4b93a6a7]) successfully deleted
I20260812 00:11:48.464522 4326 raft_consensus.cc:482] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:48.464638 4326 raft_consensus.cc:504] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.464850 4326 leader_election.cc:288] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303), 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:48.465193 3742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d21570e0214f7f9605d73b80c58630" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" is_pre_election: true
I20260812 00:11:48.465193 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d21570e0214f7f9605d73b80c58630" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784" is_pre_election: true
I20260812 00:11:48.465317 3742 raft_consensus.cc:2454] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 0.
I20260812 00:11:48.465322 4003 raft_consensus.cc:2454] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 0.
I20260812 00:11:48.465503 3356 leader_election.cc:302] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6435b61b6cc4f61abc372644ef97b13, ff533b3951cc4dc9b6cc22e05c32daa0; no voters:
I20260812 00:11:48.465682 4326 raft_consensus.cc:2781] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:48.465734 4326 raft_consensus.cc:482] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:48.465749 4326 raft_consensus.cc:3037] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:48.466452 4326 raft_consensus.cc:504] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.466602 4326 leader_election.cc:288] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 election: Requested vote from peers ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303), 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101)
I20260812 00:11:48.466786 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d21570e0214f7f9605d73b80c58630" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639d2464e36d412e8a916d486b299784"
I20260812 00:11:48.466779 3742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d21570e0214f7f9605d73b80c58630" candidate_uuid: "b6435b61b6cc4f61abc372644ef97b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0"
I20260812 00:11:48.466864 3742 raft_consensus.cc:3037] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:48.466864 4003 raft_consensus.cc:3037] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:48.467636 4003 raft_consensus.cc:2454] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 1.
I20260812 00:11:48.467636 3742 raft_consensus.cc:2454] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6435b61b6cc4f61abc372644ef97b13 in term 1.
I20260812 00:11:48.467840 3355 leader_election.cc:302] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639d2464e36d412e8a916d486b299784, b6435b61b6cc4f61abc372644ef97b13; no voters:
I20260812 00:11:48.468014 4326 raft_consensus.cc:2781] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:48.468163 4326 raft_consensus.cc:686] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [term 1 LEADER]: Becoming Leader. State: Replica: b6435b61b6cc4f61abc372644ef97b13, State: Running, Role: LEADER
I20260812 00:11:48.468227 4326 consensus_queue.cc:237] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } }
I20260812 00:11:48.468925 3191 catalog_manager.cc:5495] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "b6435b61b6cc4f61abc372644ef97b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6435b61b6cc4f61abc372644ef97b13" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 } health_report { overall_health: UNKNOWN } } }
I20260812 00:11:48.476043 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:48.473834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.31.227.1:44469)
0812 00:11:48.474008 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:48.474010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:48.474017 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:11:48.474037 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:48.474038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:48.474048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:48.474107 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:48.474246 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:48.474734 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:48.475446 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:48.475631 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:48.475679 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:48.475802 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:48.475829 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:48.475867 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:48.475923 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 00:11:48.475940 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":117,"threads_started":1}
I20260812 00:11:48.478264 3742 raft_consensus.cc:1261] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Refusing update from remote peer b6435b61b6cc4f61abc372644ef97b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:48.478392 4003 raft_consensus.cc:1261] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Refusing update from remote peer b6435b61b6cc4f61abc372644ef97b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:48.478464 4342 consensus_queue.cc:1035] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff533b3951cc4dc9b6cc22e05c32daa0" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:48.478590 4326 consensus_queue.cc:1035] T 13d21570e0214f7f9605d73b80c58630 P b6435b61b6cc4f61abc372644ef97b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639d2464e36d412e8a916d486b299784" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:11:48.482666 4258 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 00:11:50.002575 3191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:50.002704 3191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:50.003645 3191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a63eb1ecaaaf4d6bb4d5948cdfce5f47: Sending DeleteTablet for 3 replicas of tablet 13d21570e0214f7f9605d73b80c58630
I20260812 00:11:50.003932 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d21570e0214f7f9605d73b80c58630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:50 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 00:11:50.003966 3700 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d21570e0214f7f9605d73b80c58630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:50 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:51824
I20260812 00:11:50.004101 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3325
I20260812 00:11:50.004179 3970 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d21570e0214f7f9605d73b80c58630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:50 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:53690
I20260812 00:11:50.010687 4392 tablet_replica.cc:330] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: stopping tablet replica
I20260812 00:11:50.010799 4392 raft_consensus.cc:2229] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:50.010871 4392 raft_consensus.cc:2258] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:50.012389 4392 ts_tablet_manager.cc:1905] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:50.014699 4392 ts_tablet_manager.cc:1918] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:11:50.014776 4392 log.cc:1196] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/13d21570e0214f7f9605d73b80c58630
I20260812 00:11:50.015007 4392 ts_tablet_manager.cc:1939] T 13d21570e0214f7f9605d73b80c58630 P 639d2464e36d412e8a916d486b299784: Deleting consensus metadata
I20260812 00:11:50.015395 3170 catalog_manager.cc:4841] TS 639d2464e36d412e8a916d486b299784 (127.31.227.3:43101): tablet 13d21570e0214f7f9605d73b80c58630 (table test-table [id=f2dc629cd40043c5a7a1a896330e5f68]) successfully deleted
I20260812 00:11:50.017925 4386 tablet_replica.cc:330] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: stopping tablet replica
I20260812 00:11:50.018023 4386 raft_consensus.cc:2229] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:50.018100 4386 raft_consensus.cc:2258] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:50.018368 4386 ts_tablet_manager.cc:1905] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:50.019626 4386 ts_tablet_manager.cc:1918] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:11:50.019706 4386 log.cc:1196] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/13d21570e0214f7f9605d73b80c58630
I20260812 00:11:50.019941 4386 ts_tablet_manager.cc:1939] T 13d21570e0214f7f9605d73b80c58630 P ff533b3951cc4dc9b6cc22e05c32daa0: Deleting consensus metadata
I20260812 00:11:50.020318 3172 catalog_manager.cc:4841] TS ff533b3951cc4dc9b6cc22e05c32daa0 (127.31.227.2:39303): tablet 13d21570e0214f7f9605d73b80c58630 (table test-table [id=f2dc629cd40043c5a7a1a896330e5f68]) successfully deleted
W20260812 00:11:50.020855 3169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:44469 (error 108)
I20260812 00:11:50.021162 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3573
I20260812 00:11:50.032608 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:50.021331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44469 (local address 127.0.0.1:50966)
0812 00:11:50.032476 (+ 11145us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:50.032558 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:44469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11106,"thread_start_us":88,"threads_started":1}
W20260812 00:11:50.032980 3169 catalog_manager.cc:4584] TS b6435b61b6cc4f61abc372644ef97b13 (127.31.227.1:44469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13d21570e0214f7f9605d73b80c58630: Network error: Client connection negotiation failed: client connection to 127.31.227.1:44469: connect: Connection refused (error 111)
I20260812 00:11:50.049705 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3850
I20260812 00:11:50.061198 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3139
2026-08-12T00:11:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5869 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:50 dist-test-slave-zr1t krb5kdc[4401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:50 dist-test-slave-zr1t krb5kdc[4401](info): set up 2 sockets
Aug 12 00:11:50 dist-test-slave-zr1t krb5kdc[4401](info): commencing operation
krb5kdc: starting...
W20260812 00:11:51.347853 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:51.474381 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34157
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:51.559616 4424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:51.559834 4424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:51.559857 4424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:51.561056 4424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:51.561100 4424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:51.561115 4424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:51.561126 4424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:51.561138 4424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:51.562716 4424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:51.563005 4424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:51.563248 4424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:51.568591 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:51.568658 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:51.568701 4424 server_base.cc:1034] running on GCE node
W20260812 00:11:51.568609 4434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:51.569053 4424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:51.569372 4424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:51.570557 4424 hybrid_clock.cc:648] HybridClock initialized: now 1786493511570536 us; error 44 us; skew 500 ppm
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:51.576696 4424 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:51.577131 4424 webserver.cc:460] Webserver started at http://127.31.227.62:39625/ using document root <none> and password file <none>
I20260812 00:11:51.577334 4424 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:51.577376 4424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:51.577478 4424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:51.578632 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "e7568eb7f2b9441fa960d674327c86d2"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "8f856639e773b5955ef8fb18b3d548cb"
server_key_iv: "1f2bec57e1e252ebb6ad7d1a2b00993a"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.578939 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "e7568eb7f2b9441fa960d674327c86d2"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "8f856639e773b5955ef8fb18b3d548cb"
server_key_iv: "1f2bec57e1e252ebb6ad7d1a2b00993a"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.580269 4424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:11:51.581003 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.581200 4424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:51.581292 4424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "e7568eb7f2b9441fa960d674327c86d2"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "8f856639e773b5955ef8fb18b3d548cb"
server_key_iv: "1f2bec57e1e252ebb6ad7d1a2b00993a"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.581353 4424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:51.596837 4424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:51.598886 4424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:51.599014 4424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:51.603004 4424 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34157
I20260812 00:11:51.603065 4548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34157 every 8 connection(s)
I20260812 00:11:51.603428 4424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:51.604408 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:51.606876 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2: Bootstrap starting.
I20260812 00:11:51.607493 4549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:51.607821 4549 log.cc:824] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:51.608558 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2: No bootstrap required, opened a new log
I20260812 00:11:51.610123 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4424
I20260812 00:11:51.610227 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:51.610421 32652 external_mini_cluster.cc:1428] Setting key a5af4c13cd599fbf74d2d13299ff62e1
I20260812 00:11:51.610519 4549 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7568eb7f2b9441fa960d674327c86d2" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34157 } }
I20260812 00:11:51.610700 4549 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:51.610728 4549 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7568eb7f2b9441fa960d674327c86d2, State: Initialized, Role: FOLLOWER
I20260812 00:11:51.610924 4549 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7568eb7f2b9441fa960d674327c86d2" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34157 } }
I20260812 00:11:51.611040 4549 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:51.611074 4549 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:51.611161 4549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:51.612218 4549 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7568eb7f2b9441fa960d674327c86d2" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34157 } }
I20260812 00:11:51.612352 4549 leader_election.cc:302] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7568eb7f2b9441fa960d674327c86d2; no voters:
I20260812 00:11:51.612569 4549 leader_election.cc:288] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:51.612601 4554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:51.612722 4554 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [term 1 LEADER]: Becoming Leader. State: Replica: e7568eb7f2b9441fa960d674327c86d2, State: Running, Role: LEADER
I20260812 00:11:51.612874 4554 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7568eb7f2b9441fa960d674327c86d2" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34157 } }
I20260812 00:11:51.612885 4549 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:51.613649 4562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7568eb7f2b9441fa960d674327c86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7568eb7f2b9441fa960d674327c86d2" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34157 } } }
I20260812 00:11:51.613756 4562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:51.614050 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7568eb7f2b9441fa960d674327c86d2. Latest consensus state: current_term: 1 leader_uuid: "e7568eb7f2b9441fa960d674327c86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7568eb7f2b9441fa960d674327c86d2" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34157 } } }
I20260812 00:11:51.614107 4578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:51.614288 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:51.614737 4578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:11:51.616293 4578 catalog_manager.cc:1347] Generated new cluster ID: 91c332a5bf3b4b04bcd8d8aaf166dc6b
I20260812 00:11:51.616350 4578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:51.622726 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:51.611057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.31.227.62:34157)
0812 00:11:51.611262 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:51.611272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:51.611312 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:11:51.611458 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:51.611464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:51.611525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:51.611675 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:51.612355 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.613713 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.614780 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.615245 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.615386 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:51.618370 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:51.618375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:51.618375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:51.618389 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:51.620433 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:51.620867 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:51.620871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:51.620871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:51.620998 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:51.621211 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:51.621211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:51.621212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:51.621552 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:51.621712 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:51.622267 (+ 555us) server_negotiation.cc:299] Negotiation successful
0812 00:11:51.622441 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260812 00:11:51.644057 4578 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:51.644639 4578 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:51.649573 4578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7568eb7f2b9441fa960d674327c86d2: Generated new TSK 0
I20260812 00:11:51.649788 4578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:51.681895 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34157
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:51.783011 4614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:51.783233 4614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:51.783259 4614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:51.784663 4614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:51.784709 4614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:51.784736 4614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:51.786309 4614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:51.786604 4614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:51.786860 4614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:51.792026 4620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:51.792167 4614 server_base.cc:1034] running on GCE node
W20260812 00:11:51.792042 4621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:51.792227 4625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:51.792461 4614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:51.792717 4614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:51.793864 4614 hybrid_clock.cc:648] HybridClock initialized: now 1786493511793842 us; error 38 us; skew 500 ppm
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:51.796311 4614 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:51.796767 4614 webserver.cc:460] Webserver started at http://127.31.227.1:45469/ using document root <none> and password file <none>
I20260812 00:11:51.797008 4614 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:51.797078 4614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:51.797231 4614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:51.798380 4614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "ca2380957a734ca4ab1491f85b8cd8ff"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "533730e6db187b8e2aa4781ba4d8ec85"
server_key_iv: "539e95f3903fa5f51e1dacfc8c13865d"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.798768 4614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "ca2380957a734ca4ab1491f85b8cd8ff"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "533730e6db187b8e2aa4781ba4d8ec85"
server_key_iv: "539e95f3903fa5f51e1dacfc8c13865d"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.800149 4614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:11:51.800900 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.801066 4614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:11:51.801180 4614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "ca2380957a734ca4ab1491f85b8cd8ff"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "533730e6db187b8e2aa4781ba4d8ec85"
server_key_iv: "539e95f3903fa5f51e1dacfc8c13865d"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.801260 4614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:51.817062 4614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:51.819470 4614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:51.819669 4614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:51.819937 4614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:51.820293 4614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:51.820361 4614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.820405 4614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:51.820457 4614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.826853 4614 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:46623
I20260812 00:11:51.826901 4848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:46623 every 8 connection(s)
I20260812 00:11:51.827244 4614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:51.829164 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4614
I20260812 00:11:51.829277 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:51.829457 32652 external_mini_cluster.cc:1428] Setting key 791d1accf13251a4008e52318ef2c6af
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:51.838255 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:51.828007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:40907 (local address 127.31.227.62:34157)
0812 00:11:51.828094 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:51.828096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:51.828351 (+ 255us) server_negotiation.cc:407] Connection header received
0812 00:11:51.828834 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:51.828835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:51.828851 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:51.828904 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:51.830020 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.830606 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.831696 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.831925 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.832008 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:51.834900 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:51.834902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:51.834902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:51.834936 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:51.836504 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:51.837078 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:51.837079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:51.837079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:51.837132 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:51.837459 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:51.837460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:51.837460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:51.837659 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:51.837715 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:51.838117 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 00:11:51.838188 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:11:51.838547 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:51.827961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34157 (local address 127.31.227.1:40907)
0812 00:11:51.828180 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:51.828247 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:51.828746 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:51.828986 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:51.828990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:51.829724 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:51.829928 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:51.829933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.830739 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.830740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:51.831582 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:51.831585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.832022 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.832023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:51.832168 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:51.832759 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:51.832768 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:51.834803 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:51.836598 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:51.836606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:51.836609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:51.836990 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:51.837204 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:51.837205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:51.837205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:51.837372 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:51.837751 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:51.837760 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:51.837960 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 00:11:51.838096 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 00:11:51.838281 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
I20260812 00:11:51.839164 4849 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34157
I20260812 00:11:51.839265 4849 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:51.839475 4849 heartbeater.cc:511] Master 127.31.227.62:34157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
I20260812 00:11:51.840088 4485 ts_manager.cc:194] Registered new tserver with Master: ca2380957a734ca4ab1491f85b8cd8ff (127.31.227.1:46623)
I20260812 00:11:51.840725 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:40907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:51.856076 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34157
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:51.941349 4860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:51.941644 4860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:51.941685 4860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:51.942898 4860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:51.942979 4860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:51.943024 4860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:51.944494 4860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:51.944773 4860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:51.945024 4860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:51.950362 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:51.950459 4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:51.950410 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:51.950484 4860 server_base.cc:1034] running on GCE node
I20260812 00:11:51.950800 4860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:51.951072 4860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:51.952247 4860 hybrid_clock.cc:648] HybridClock initialized: now 1786493511952235 us; error 30 us; skew 500 ppm
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:51.955041 4860 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:51.955577 4860 webserver.cc:460] Webserver started at http://127.31.227.2:37385/ using document root <none> and password file <none>
I20260812 00:11:51.955922 4860 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:51.956003 4860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:51.956161 4860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:51.957763 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "c5435b48d7eb4057ac4fdb316dfd2340"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "690b87e8d6c97a28f8537b25ae0a7464"
server_key_iv: "7c2648eabe2746cd7f2451186a249e89"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.958259 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "c5435b48d7eb4057ac4fdb316dfd2340"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "690b87e8d6c97a28f8537b25ae0a7464"
server_key_iv: "7c2648eabe2746cd7f2451186a249e89"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.960088 4860 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 00:11:51.960927 4897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.961241 4860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:51.961359 4860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "c5435b48d7eb4057ac4fdb316dfd2340"
format_stamp: "Formatted at 2026-08-12 00:11:51 on dist-test-slave-zr1t"
server_key: "690b87e8d6c97a28f8537b25ae0a7464"
server_key_iv: "7c2648eabe2746cd7f2451186a249e89"
server_key_version: "encryptionkey@0"
I20260812 00:11:51.961440 4860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:51.971185 4860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:51.973237 4860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:51.973417 4860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:51.973717 4860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:51.974072 4860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:51.974131 4860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.974167 4860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:51.974213 4860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:51.980469 4860 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:40749
I20260812 00:11:51.980512 5118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:40749 every 8 connection(s)
I20260812 00:11:51.980844 4860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:51.981791 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4860
I20260812 00:11:51.981899 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:51.982133 32652 external_mini_cluster.cc:1428] Setting key 4321adc2fce35002d279510f84205e4e
Aug 12 00:11:51 dist-test-slave-zr1t krb5kdc[4401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493511, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:51.991868 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:51.981615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:35773 (local address 127.31.227.62:34157)
0812 00:11:51.981695 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:51.981698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:51.981983 (+ 285us) server_negotiation.cc:407] Connection header received
0812 00:11:51.982658 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:51.982659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:51.982670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:51.982732 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:51.984186 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.984752 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.985965 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.986163 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.986255 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:51.988853 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:51.988855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:51.988855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:51.988870 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:51.990374 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:51.990847 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:51.990848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:51.990848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:51.990912 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:51.991188 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:51.991189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:51.991190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:51.991314 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:51.991395 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:51.991685 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 00:11:51.991776 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
I20260812 00:11:51.992080 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:51.981591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34157 (local address 127.31.227.2:35773)
0812 00:11:51.981816 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:51.981873 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:51.982560 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:51.982846 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:51.982851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:51.983773 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:51.984081 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:51.984088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.984860 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.984861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:51.985869 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:51.985872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:51.986282 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:51.986283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:51.986436 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:51.987035 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:51.987046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:51.988742 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:51.990491 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:51.990499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:51.990501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:51.990747 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:51.990996 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:51.990997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:51.990997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:51.991111 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:51.991387 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:51.991395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:51.991543 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 00:11:51.991652 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:11:51.991820 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260812 00:11:51.992826 5119 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34157
I20260812 00:11:51.992947 5119 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:51.993137 5119 heartbeater.cc:511] Master 127.31.227.62:34157 requested a full tablet report, sending...
I20260812 00:11:51.993587 4485 ts_manager.cc:194] Registered new tserver with Master: c5435b48d7eb4057ac4fdb316dfd2340 (127.31.227.2:40749)
I20260812 00:11:51.993961 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:35773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:52.010185 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34157
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:52.100991 5126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:52.101233 5126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:52.101272 5126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:52.102540 5126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:52.102609 5126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:52.102655 5126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:52.104306 5126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:52.104583 5126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:52.104902 5126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:52.110121 5135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:52.110149 5126 server_base.cc:1034] running on GCE node
W20260812 00:11:52.110173 5138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:52.110121 5136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:52.110608 5126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:52.110884 5126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:52.112054 5126 hybrid_clock.cc:648] HybridClock initialized: now 1786493512112028 us; error 37 us; skew 500 ppm
Aug 12 00:11:52 dist-test-slave-zr1t krb5kdc[4401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493512, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:52.115255 5126 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:52.115720 5126 webserver.cc:460] Webserver started at http://127.31.227.3:36639/ using document root <none> and password file <none>
I20260812 00:11:52.115914 5126 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:52.115955 5126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:52.116055 5126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:52.117095 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "41a4c4f361434f779ee8450e998c5949"
format_stamp: "Formatted at 2026-08-12 00:11:52 on dist-test-slave-zr1t"
server_key: "be5812ac8fc8cbceb2a0c4b0b267239c"
server_key_iv: "18ce7b1ff36412106200e7df0bea3ef2"
server_key_version: "encryptionkey@0"
I20260812 00:11:52.117409 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "41a4c4f361434f779ee8450e998c5949"
format_stamp: "Formatted at 2026-08-12 00:11:52 on dist-test-slave-zr1t"
server_key: "be5812ac8fc8cbceb2a0c4b0b267239c"
server_key_iv: "18ce7b1ff36412106200e7df0bea3ef2"
server_key_version: "encryptionkey@0"
I20260812 00:11:52.118892 5126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:52.119601 5146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:52.119777 5126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:52.119866 5126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "41a4c4f361434f779ee8450e998c5949"
format_stamp: "Formatted at 2026-08-12 00:11:52 on dist-test-slave-zr1t"
server_key: "be5812ac8fc8cbceb2a0c4b0b267239c"
server_key_iv: "18ce7b1ff36412106200e7df0bea3ef2"
server_key_version: "encryptionkey@0"
I20260812 00:11:52.119926 5126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:52.126889 5126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:52.128822 5126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:52.128937 5126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:52.129125 5126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:52.129459 5126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:52.129495 5126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:52.129585 5126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:52.129632 5126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:52.135680 5126 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:45929
I20260812 00:11:52.135766 5354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:45929 every 8 connection(s)
I20260812 00:11:52.136106 5126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:11:52 dist-test-slave-zr1t krb5kdc[4401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493512, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:52.145462 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:52.136864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:51725 (local address 127.31.227.62:34157)
0812 00:11:52.136944 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:52.136946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:52.137135 (+ 189us) server_negotiation.cc:407] Connection header received
0812 00:11:52.137709 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:52.137711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:52.137722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:52.137776 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:52.138724 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:52.139215 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:52.140190 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:52.140371 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:52.140422 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:52.142768 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:52.142770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:52.142770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:52.142778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:52.144079 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:52.144500 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:52.144501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:52.144501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:52.144554 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:52.144777 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:52.144777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:52.144777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:52.144885 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:52.144937 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:52.145289 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 00:11:52.145378 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:11:52.145663 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:52.136812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34157 (local address 127.31.227.3:51725)
0812 00:11:52.137008 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:52.137046 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:52.137608 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:52.137833 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:52.137836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:52.138438 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:52.138645 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:52.138650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:52.139300 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:52.139301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:52.140114 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:52.140116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:52.140471 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:52.140472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:52.140607 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:52.141103 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:52.141112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:52.142603 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:52.144157 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:52.144164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:52.144166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:52.144416 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:52.144615 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:52.144616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:52.144616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:52.144721 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:52.144968 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:52.144976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:52.145110 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 00:11:52.145218 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:11:52.145354 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260812 00:11:52.146188 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5126
I20260812 00:11:52.146281 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:52.146235 5356 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34157
I20260812 00:11:52.146380 5356 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:52.146510 32652 external_mini_cluster.cc:1428] Setting key 94723886a5e2e1e4988aee9a984d09b6
I20260812 00:11:52.146636 5356 heartbeater.cc:511] Master 127.31.227.62:34157 requested a full tablet report, sending...
I20260812 00:11:52.147039 4485 ts_manager.cc:194] Registered new tserver with Master: 41a4c4f361434f779ee8450e998c5949 (127.31.227.3:45929)
I20260812 00:11:52.147477 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:51725
I20260812 00:11:52.147657 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:11:52.161017 4553 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:11:52.160081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.31.227.62:34157)
0812 00:11:52.160169 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:52.160172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:52.160246 (+ 74us) server_negotiation.cc:407] Connection header received
0812 00:11:52.160299 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:52.160300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:52.160311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:52.160378 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:52.160907 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41142: BlockingRecv error: recv got EOF from 127.0.0.1:41142 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:11:52.162016 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4614
I20260812 00:11:52.186153 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4860
I20260812 00:11:52.206739 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5126
I20260812 00:11:52.219558 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4424
2026-08-12T00:11:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2159 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:52 dist-test-slave-zr1t krb5kdc[5376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:52 dist-test-slave-zr1t krb5kdc[5376](info): set up 2 sockets
Aug 12 00:11:52 dist-test-slave-zr1t krb5kdc[5376](info): commencing operation
krb5kdc: starting...
W20260812 00:11:53.507015 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:53 dist-test-slave-zr1t krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:53.630326 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38953
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:38953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:53.715133 5402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:53.715349 5402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:53.715371 5402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:53.716555 5402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:53.716600 5402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:53.716614 5402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:53.716655 5402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:53.716686 5402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:53.718237 5402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:38953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:53.718503 5402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:53.718726 5402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:53.723766 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:53.723798 5410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:53.723776 5409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:53.724110 5402 server_base.cc:1034] running on GCE node
I20260812 00:11:53.724392 5402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:53.724707 5402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:53.725903 5402 hybrid_clock.cc:648] HybridClock initialized: now 1786493513725877 us; error 36 us; skew 500 ppm
Aug 12 00:11:53 dist-test-slave-zr1t krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493513, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:53.728621 5402 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:53.729070 5402 webserver.cc:460] Webserver started at http://127.31.227.62:35969/ using document root <none> and password file <none>
I20260812 00:11:53.729307 5402 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:53.729357 5402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:53.729578 5402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:53.730679 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "75c2b02b513b4d5f9f525b3df1287eb4"
format_stamp: "Formatted at 2026-08-12 00:11:53 on dist-test-slave-zr1t"
server_key: "376395aee72e6f766b94c99fbe0690b9"
server_key_iv: "c444bb362c3ad8d7e9d7932b6c601d83"
server_key_version: "encryptionkey@0"
I20260812 00:11:53.731057 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "75c2b02b513b4d5f9f525b3df1287eb4"
format_stamp: "Formatted at 2026-08-12 00:11:53 on dist-test-slave-zr1t"
server_key: "376395aee72e6f766b94c99fbe0690b9"
server_key_iv: "c444bb362c3ad8d7e9d7932b6c601d83"
server_key_version: "encryptionkey@0"
I20260812 00:11:53.732442 5402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:11:53.733233 5427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:53.733575 5402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:11:53.733680 5402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "75c2b02b513b4d5f9f525b3df1287eb4"
format_stamp: "Formatted at 2026-08-12 00:11:53 on dist-test-slave-zr1t"
server_key: "376395aee72e6f766b94c99fbe0690b9"
server_key_iv: "c444bb362c3ad8d7e9d7932b6c601d83"
server_key_version: "encryptionkey@0"
I20260812 00:11:53.733737 5402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:53.754135 5402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:53.756345 5402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:53.756462 5402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:53.760603 5402 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:38953
I20260812 00:11:53.760679 5530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:38953 every 8 connection(s)
I20260812 00:11:53.761229 5402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:53.761960 5532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:53.764372 5532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4: Bootstrap starting.
I20260812 00:11:53.764984 5532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:53.765311 5532 log.cc:824] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:53.765802 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5402
I20260812 00:11:53.765911 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:53.766099 32652 external_mini_cluster.cc:1428] Setting key 1d49bf84cd04455c41bee3b5942cba93
I20260812 00:11:53.766152 5532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4: No bootstrap required, opened a new log
I20260812 00:11:53.768445 5532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38953 } }
I20260812 00:11:53.768611 5532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:53.768678 5532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c2b02b513b4d5f9f525b3df1287eb4, State: Initialized, Role: FOLLOWER
I20260812 00:11:53.768837 5532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38953 } }
I20260812 00:11:53.768906 5532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:53.768929 5532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:53.769006 5532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:53.769593 5532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38953 } }
I20260812 00:11:53.769719 5532 leader_election.cc:302] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75c2b02b513b4d5f9f525b3df1287eb4; no voters:
I20260812 00:11:53.769907 5532 leader_election.cc:288] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:53.769970 5546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:53.770147 5546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 75c2b02b513b4d5f9f525b3df1287eb4, State: Running, Role: LEADER
I20260812 00:11:53.770176 5532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:53.770298 5546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38953 } }
I20260812 00:11:53.770792 5553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75c2b02b513b4d5f9f525b3df1287eb4. Latest consensus state: current_term: 1 leader_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38953 } } }
I20260812 00:11:53.770876 5553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:53.771119 5547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2b02b513b4d5f9f525b3df1287eb4" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38953 } } }
I20260812 00:11:53.771461 5564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:53.772068 5564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:11:53.772804 5547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:53.774010 5564 catalog_manager.cc:1347] Generated new cluster ID: fa64bcdc809e4aed87b56ee1e99e8f23
I20260812 00:11:53.774060 5564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:11:53 dist-test-slave-zr1t krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:53.779742 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:53.766636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.31.227.62:38953)
0812 00:11:53.766832 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:53.766850 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:53.766885 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:11:53.768435 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:53.768443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:53.768493 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:53.768641 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:53.769409 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:53.770804 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:53.771897 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:53.772499 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:53.772614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:53.775559 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:53.775564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:53.775564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:53.775576 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:53.777619 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:53.778031 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:53.778034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:53.778034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:53.778162 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:53.778373 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:53.778374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:53.778374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:53.778753 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:53.778934 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:53.779381 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 00:11:53.779536 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260812 00:11:53.780705 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:11:53.790791 5564 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:53.791353 5564 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:53.794803 5564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75c2b02b513b4d5f9f525b3df1287eb4: Generated new TSK 0
I20260812 00:11:53.794994 5564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:11:53.799487 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:53.795557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.31.227.62:38953)
0812 00:11:53.795624 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:53.795626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:53.795721 (+ 95us) server_negotiation.cc:407] Connection header received
0812 00:11:53.795798 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:53.795798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:53.795807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:53.795860 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:53.796250 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:53.796743 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:53.797880 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:53.798156 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:53.798241 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:53.799004 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:53.799005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:53.799006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:11:53.799173 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:53.799189 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:11:53.799195 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:53.799250 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:53.799381 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 00:11:53.799404 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:11:53.801887 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5402
2026-08-12T00:11:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1578 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:53 dist-test-slave-zr1t krb5kdc[5602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:53 dist-test-slave-zr1t krb5kdc[5602](info): set up 2 sockets
Aug 12 00:11:53 dist-test-slave-zr1t krb5kdc[5602](info): commencing operation
krb5kdc: starting...
W20260812 00:11:55.075582 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:55.196200 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45247
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:45247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:55.278937 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:55.279136 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:55.279158 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:55.280299 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:55.280341 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:55.280356 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:55.280397 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:55.280428 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:55.281910 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:45247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:55.282159 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:55.282400 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:55.287242 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.287223 5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.287297 5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:55.287439 5626 server_base.cc:1034] running on GCE node
I20260812 00:11:55.287695 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:55.288023 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:55.289189 5626 hybrid_clock.cc:648] HybridClock initialized: now 1786493515289174 us; error 27 us; skew 500 ppm
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:55.291630 5626 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:55.292097 5626 webserver.cc:460] Webserver started at http://127.31.227.62:33435/ using document root <none> and password file <none>
I20260812 00:11:55.292316 5626 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:55.292384 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:55.292539 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:55.293640 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "23334eb544c54bf28ba50cffb4beddb3"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "69e9bcb7b5c3a2a03513390212d8c7e8"
server_key_iv: "a8910dc27240cdd1f57a14c8f132c6d8"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.294014 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "23334eb544c54bf28ba50cffb4beddb3"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "69e9bcb7b5c3a2a03513390212d8c7e8"
server_key_iv: "a8910dc27240cdd1f57a14c8f132c6d8"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.295307 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:55.296067 5650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.296237 5626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:55.296345 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "23334eb544c54bf28ba50cffb4beddb3"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "69e9bcb7b5c3a2a03513390212d8c7e8"
server_key_iv: "a8910dc27240cdd1f57a14c8f132c6d8"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.296422 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:55.308444 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:55.310253 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:55.310415 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:55.314640 5626 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:45247
I20260812 00:11:55.314673 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:45247 every 8 connection(s)
I20260812 00:11:55.315027 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:55.316073 5743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:55.318444 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3: Bootstrap starting.
I20260812 00:11:55.319023 5743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:55.319306 5743 log.cc:824] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:55.320019 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3: No bootstrap required, opened a new log
I20260812 00:11:55.321126 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5626
I20260812 00:11:55.321244 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:55.321427 32652 external_mini_cluster.cc:1428] Setting key 43c3969d9fe9888a1f39132838f2edc2
I20260812 00:11:55.322163 5743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23334eb544c54bf28ba50cffb4beddb3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45247 } }
I20260812 00:11:55.322343 5743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:55.322391 5743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23334eb544c54bf28ba50cffb4beddb3, State: Initialized, Role: FOLLOWER
I20260812 00:11:55.322543 5743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23334eb544c54bf28ba50cffb4beddb3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45247 } }
I20260812 00:11:55.322636 5743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:55.322672 5743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:55.322772 5743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:55.323523 5743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23334eb544c54bf28ba50cffb4beddb3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45247 } }
I20260812 00:11:55.323614 5743 leader_election.cc:302] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23334eb544c54bf28ba50cffb4beddb3; no voters:
I20260812 00:11:55.323799 5743 leader_election.cc:288] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:55.323854 5751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:55.324002 5751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [term 1 LEADER]: Becoming Leader. State: Replica: 23334eb544c54bf28ba50cffb4beddb3, State: Running, Role: LEADER
I20260812 00:11:55.324066 5743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:55.324157 5751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23334eb544c54bf28ba50cffb4beddb3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45247 } }
I20260812 00:11:55.324512 5751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23334eb544c54bf28ba50cffb4beddb3. Latest consensus state: current_term: 1 leader_uuid: "23334eb544c54bf28ba50cffb4beddb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23334eb544c54bf28ba50cffb4beddb3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45247 } } }
I20260812 00:11:55.324549 5752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23334eb544c54bf28ba50cffb4beddb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23334eb544c54bf28ba50cffb4beddb3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45247 } } }
I20260812 00:11:55.324630 5751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:55.324645 5752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:55.324915 5766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:55.325487 5766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:11:55.328115 5766 catalog_manager.cc:1347] Generated new cluster ID: bc293f2d404b4fd195d980ca131fa37d
I20260812 00:11:55.328194 5766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:55.332527 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.321995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.31.227.62:45247)
0812 00:11:55.322186 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:55.322201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:55.322226 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:11:55.322331 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:55.322336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:55.322400 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:55.322585 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:55.323375 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.324689 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.325801 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.326380 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.326524 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.329228 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:55.329232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:55.329232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:55.329244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:55.330973 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.331354 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.331357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.331358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.331474 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.331688 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.331689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.331689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.331941 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:55.332118 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:55.332195 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 00:11:55.332334 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260812 00:11:55.342530 5766 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:55.343065 5766 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:55.346174 5766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23334eb544c54bf28ba50cffb4beddb3: Generated new TSK 0
I20260812 00:11:55.346347 5766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:55.378428 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45247
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:55.467947 5790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:55.468174 5790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:55.468215 5790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:55.469364 5790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:55.469435 5790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:55.469480 5790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:55.470981 5790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:55.471251 5790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:55.471496 5790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:55.476748 5803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.476732 5813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.476751 5809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:55.477017 5790 server_base.cc:1034] running on GCE node
I20260812 00:11:55.477226 5790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:55.477484 5790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:55.478668 5790 hybrid_clock.cc:648] HybridClock initialized: now 1786493515478654 us; error 28 us; skew 500 ppm
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:55.481302 5790 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:55.481812 5790 webserver.cc:460] Webserver started at http://127.31.227.1:44937/ using document root <none> and password file <none>
I20260812 00:11:55.482021 5790 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:55.482062 5790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:55.482160 5790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:55.483179 5790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "92a1849b9ff243f2b40bb39d19101985"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "08830b256a5e5f51379cc759034257eb"
server_key_iv: "33c97d9836f9a3cc2d34b2da36296ce9"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.483500 5790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "92a1849b9ff243f2b40bb39d19101985"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "08830b256a5e5f51379cc759034257eb"
server_key_iv: "33c97d9836f9a3cc2d34b2da36296ce9"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.484848 5790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:11:55.485625 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.485800 5790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:11:55.485908 5790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "92a1849b9ff243f2b40bb39d19101985"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "08830b256a5e5f51379cc759034257eb"
server_key_iv: "33c97d9836f9a3cc2d34b2da36296ce9"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.485991 5790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:55.497189 5790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:55.499583 5790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:55.499748 5790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:55.499943 5790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:55.500299 5790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:55.500363 5790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.500420 5790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:55.500449 5790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.506788 5790 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:39237
I20260812 00:11:55.506834 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:39237 every 8 connection(s)
I20260812 00:11:55.507139 5790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:55.514271 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5790
I20260812 00:11:55.514391 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:55.514632 32652 external_mini_cluster.cc:1428] Setting key 22a9210f4074757b1db6ed7329687dc1
I20260812 00:11:55.516969 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.507863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:60629 (local address 127.31.227.62:45247)
0812 00:11:55.507930 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:55.507932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:55.508136 (+ 204us) server_negotiation.cc:407] Connection header received
0812 00:11:55.508689 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:55.508691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:55.508705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:55.508756 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:55.509793 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.510299 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.511368 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.511576 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.511685 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.513915 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:55.513916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:55.513916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:55.513924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:55.515509 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.516000 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.516001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.516002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.516054 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.516354 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.516354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.516355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.516455 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:55.516507 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:55.516839 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 00:11:55.516906 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:11:55.517282 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.507825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45247 (local address 127.31.227.1:60629)
0812 00:11:55.508014 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:55.508056 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:55.508576 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:55.508825 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:55.508829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:55.509407 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:55.509684 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:55.509690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.510414 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.510415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:55.511254 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:55.511256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.511710 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.511711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:55.511852 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.512306 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:55.512315 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:55.513817 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:55.515592 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:55.515601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:55.515603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:55.515921 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:55.516121 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:55.516122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:55.516122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:55.516273 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:55.516521 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:55.516528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:55.516699 (+ 171us) client_negotiation.cc:769] Sending connection context
0812 00:11:55.516817 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:11:55.517000 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260812 00:11:55.517982 6059 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45247
I20260812 00:11:55.518095 6059 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:55.518296 6059 heartbeater.cc:511] Master 127.31.227.62:45247 requested a full tablet report, sending...
I20260812 00:11:55.518853 5685 ts_manager.cc:194] Registered new tserver with Master: 92a1849b9ff243f2b40bb39d19101985 (127.31.227.1:39237)
I20260812 00:11:55.519472 5685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:60629
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:55.540051 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45247
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:55.622356 6069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:55.622583 6069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:55.622624 6069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:55.623826 6069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:55.623894 6069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:55.623940 6069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:55.625692 6069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:55.625957 6069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:55.626194 6069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:55.631326 6079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.631300 6075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.631304 6086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:55.631407 6069 server_base.cc:1034] running on GCE node
I20260812 00:11:55.631768 6069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:55.632016 6069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:55.633177 6069 hybrid_clock.cc:648] HybridClock initialized: now 1786493515633161 us; error 29 us; skew 500 ppm
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:55.635712 6069 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:55.636185 6069 webserver.cc:460] Webserver started at http://127.31.227.2:39957/ using document root <none> and password file <none>
I20260812 00:11:55.636410 6069 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:55.636476 6069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:55.636629 6069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:55.637741 6069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "cf1b75c4eba945bebba6bc22a1686b80"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "cd029a78e4f77c65ab94ec111008b829"
server_key_iv: "eaf1e553ef671811e3adf041db57e215"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.638113 6069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "cf1b75c4eba945bebba6bc22a1686b80"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "cd029a78e4f77c65ab94ec111008b829"
server_key_iv: "eaf1e553ef671811e3adf041db57e215"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.639446 6069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:55.640213 6109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.640389 6069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:55.640498 6069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "cf1b75c4eba945bebba6bc22a1686b80"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "cd029a78e4f77c65ab94ec111008b829"
server_key_iv: "eaf1e553ef671811e3adf041db57e215"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.640573 6069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:55.650940 6069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:55.652669 6069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:55.652825 6069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:55.653049 6069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:55.653358 6069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:55.653414 6069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.653456 6069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:55.653484 6069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.659662 6069 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:38211
I20260812 00:11:55.659772 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:38211 every 8 connection(s)
I20260812 00:11:55.660076 6069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:55.664853 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6069
I20260812 00:11:55.664983 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:55.665195 32652 external_mini_cluster.cc:1428] Setting key e728b052cedd564f81bec63b3a229203
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:55.670753 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.660901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:44847 (local address 127.31.227.62:45247)
0812 00:11:55.660972 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:55.660975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:55.661174 (+ 199us) server_negotiation.cc:407] Connection header received
0812 00:11:55.661798 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:55.661799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:55.661813 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:55.661888 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:55.662875 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.663379 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.664417 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.664619 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.664701 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.667365 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:55.667367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:55.667367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:55.667375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:55.669125 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.669696 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.669697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.669697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.669782 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.670112 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.670113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.670113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.670221 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:55.670272 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:55.670614 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 00:11:55.670688 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:11:55.671059 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.660874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45247 (local address 127.31.227.2:44847)
0812 00:11:55.661051 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:55.661088 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:55.661687 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:55.661960 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:55.661965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:55.662573 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:55.662762 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:55.662767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.663479 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.663481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:55.664296 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:55.664298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.664721 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.664722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:55.664848 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.665420 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:55.665431 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:55.667267 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:55.669205 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:55.669239 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:55.669242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:55.669605 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:55.669870 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:55.669871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:55.669872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:55.670036 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:55.670287 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:55.670294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:55.670466 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 00:11:55.670584 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:11:55.670794 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260812 00:11:55.671563 6336 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45247
I20260812 00:11:55.671674 6336 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:55.671850 6336 heartbeater.cc:511] Master 127.31.227.62:45247 requested a full tablet report, sending...
I20260812 00:11:55.672247 5685 ts_manager.cc:194] Registered new tserver with Master: cf1b75c4eba945bebba6bc22a1686b80 (127.31.227.2:38211)
I20260812 00:11:55.672798 5685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:44847
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:55.692010 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45247
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:55.775449 6347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:55.775636 6347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:55.775660 6347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:55.776800 6347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:55.776841 6347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:55.776872 6347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:55.778373 6347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:55.778623 6347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:55.778880 6347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:55.784168 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:55.784190 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:55.784297 6347 server_base.cc:1034] running on GCE node
W20260812 00:11:55.784436 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:55.784698 6347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:55.784947 6347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:55.786104 6347 hybrid_clock.cc:648] HybridClock initialized: now 1786493515786098 us; error 38 us; skew 500 ppm
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:55.792671 6347 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:55.793114 6347 webserver.cc:460] Webserver started at http://127.31.227.3:46157/ using document root <none> and password file <none>
I20260812 00:11:55.793349 6347 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:55.793416 6347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:55.793597 6347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:55.794638 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "e15ad2af3f304a35beecd5dce07cea01"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "ec416a591b519e2364055ecc54d580da"
server_key_iv: "85c686d417e9fd03f74e3088b1a99a40"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.795013 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "e15ad2af3f304a35beecd5dce07cea01"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "ec416a591b519e2364055ecc54d580da"
server_key_iv: "85c686d417e9fd03f74e3088b1a99a40"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.796403 6347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:55.797230 6377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.797421 6347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:55.797523 6347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "e15ad2af3f304a35beecd5dce07cea01"
format_stamp: "Formatted at 2026-08-12 00:11:55 on dist-test-slave-zr1t"
server_key: "ec416a591b519e2364055ecc54d580da"
server_key_iv: "85c686d417e9fd03f74e3088b1a99a40"
server_key_version: "encryptionkey@0"
I20260812 00:11:55.797637 6347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:55.806820 6347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:55.808687 6347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:55.808848 6347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:55.809074 6347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:55.809387 6347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:55.809449 6347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.809482 6347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:55.809510 6347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:55.815747 6347 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:34549
I20260812 00:11:55.815797 6586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:34549 every 8 connection(s)
I20260812 00:11:55.816092 6347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:11:55.817019 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6347
I20260812 00:11:55.817130 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:55.817301 32652 external_mini_cluster.cc:1428] Setting key c66b4073317bb4094e2f74e67effaaf0
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:55.826318 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.816771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:38919 (local address 127.31.227.62:45247)
0812 00:11:55.816853 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:55.816855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:55.817643 (+ 788us) server_negotiation.cc:407] Connection header received
0812 00:11:55.818327 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:55.818328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:55.818340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:55.818391 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:55.819307 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.819831 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.820859 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.821051 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.821117 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.823362 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:55.823364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:55.823364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:55.823371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:55.824808 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.825235 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.825236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.825237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.825306 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.825619 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.825620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.825620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.825745 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:55.825853 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:55.826143 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 00:11:55.826230 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:11:55.826508 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.816726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45247 (local address 127.31.227.3:38919)
0812 00:11:55.817485 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:55.817565 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:55.818216 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:55.818464 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:55.818468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:55.819026 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:55.819209 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:55.819214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.819925 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.819926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:55.820749 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:55.820751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.821142 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.821142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:55.821250 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.821852 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:55.821861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:55.823255 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:55.824871 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:55.824878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:55.824880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:55.825152 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:55.825379 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:55.825380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:55.825380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:55.825496 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:55.825845 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:55.825853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:55.825999 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 00:11:55.826104 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:11:55.826262 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":87,"threads_started":1}
I20260812 00:11:55.827020 6587 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45247
I20260812 00:11:55.827153 6587 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:55.827335 6587 heartbeater.cc:511] Master 127.31.227.62:45247 requested a full tablet report, sending...
I20260812 00:11:55.827787 5685 ts_manager.cc:194] Registered new tserver with Master: e15ad2af3f304a35beecd5dce07cea01 (127.31.227.3:34549)
I20260812 00:11:55.828177 5685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:38919
I20260812 00:11:55.829167 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:55.853289 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:55.846294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.31.227.62:45247)
0812 00:11:55.846367 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:55.846374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:55.846451 (+ 77us) server_negotiation.cc:407] Connection header received
0812 00:11:55.846516 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:55.846516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:55.846525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:55.846567 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:55.847207 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.847686 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.848452 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:55.848632 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:55.848683 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:55.851036 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:55.851037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:55.851037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:55.851044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:55.852267 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.852570 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.852571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.852571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.852620 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:55.852779 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:55.852779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:55.852779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:55.852914 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:55.852988 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:55.853103 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 00:11:55.853171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 00:11:55.853603 5685 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34486
I20260812 00:11:55.854208 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5790
I20260812 00:11:55.866775 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6069
I20260812 00:11:55.878463 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6347
I20260812 00:11:55.889461 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5626
2026-08-12T00:11:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2084 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[6603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[6603](info): set up 2 sockets
Aug 12 00:11:55 dist-test-slave-zr1t krb5kdc[6603](info): commencing operation
krb5kdc: starting...
W20260812 00:11:57.185984 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:57.309752 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:35363
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:57.393338 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:57.393625 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:57.393668 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:57.394932 6627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:57.395009 6627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:57.395033 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:57.395053 6627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:57.395084 6627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:57.396683 6627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:57.396981 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:57.397233 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:57.402366 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.402366 6635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.402439 6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:57.402828 6627 server_base.cc:1034] running on GCE node
I20260812 00:11:57.403095 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:57.403338 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:57.404487 6627 hybrid_clock.cc:648] HybridClock initialized: now 1786493517404473 us; error 32 us; skew 500 ppm
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:57.410305 6627 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:57.410818 6627 webserver.cc:460] Webserver started at http://127.31.227.62:39489/ using document root <none> and password file <none>
I20260812 00:11:57.411077 6627 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:57.411147 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:57.411304 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:57.412396 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "add71c01b7fb4ec8aac40fdd4a31052f"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "a84634d78ac22d110112101beb9989a5"
server_key_iv: "d888522093cb5a78055982964b631726"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.412789 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "add71c01b7fb4ec8aac40fdd4a31052f"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "a84634d78ac22d110112101beb9989a5"
server_key_iv: "d888522093cb5a78055982964b631726"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.414238 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:11:57.415062 6655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.415261 6627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:57.415374 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "add71c01b7fb4ec8aac40fdd4a31052f"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "a84634d78ac22d110112101beb9989a5"
server_key_iv: "d888522093cb5a78055982964b631726"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.415452 6627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:57.430842 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:57.433012 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:57.433216 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:57.437831 6627 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:35363
I20260812 00:11:57.437885 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:35363 every 8 connection(s)
I20260812 00:11:57.438443 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:57.439162 6752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:57.441474 6752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f: Bootstrap starting.
I20260812 00:11:57.442129 6752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:57.442414 6752 log.cc:824] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:57.443120 6752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f: No bootstrap required, opened a new log
I20260812 00:11:57.444691 6752 raft_consensus.cc:348] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35363 } }
I20260812 00:11:57.444903 6752 raft_consensus.cc:374] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:57.444950 6752 raft_consensus.cc:729] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add71c01b7fb4ec8aac40fdd4a31052f, State: Initialized, Role: FOLLOWER
I20260812 00:11:57.445003 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6627
I20260812 00:11:57.445111 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:57.445106 6752 consensus_queue.cc:260] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35363 } }
I20260812 00:11:57.445189 6752 raft_consensus.cc:388] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:57.445230 6752 raft_consensus.cc:482] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:57.445286 6752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:57.445327 32652 external_mini_cluster.cc:1428] Setting key 826c1efda0e8073b2b383a31c1b3a38f
I20260812 00:11:57.446554 6752 raft_consensus.cc:504] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35363 } }
I20260812 00:11:57.446681 6752 leader_election.cc:302] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: add71c01b7fb4ec8aac40fdd4a31052f; no voters:
I20260812 00:11:57.446861 6752 leader_election.cc:288] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:57.446938 6760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:57.447078 6760 raft_consensus.cc:686] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [term 1 LEADER]: Becoming Leader. State: Replica: add71c01b7fb4ec8aac40fdd4a31052f, State: Running, Role: LEADER
I20260812 00:11:57.447170 6760 consensus_queue.cc:237] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35363 } }
I20260812 00:11:57.447180 6752 sys_catalog.cc:564] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:57.448439 6764 sys_catalog.cc:455] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [sys.catalog]: SysCatalogTable state changed. Reason: New leader add71c01b7fb4ec8aac40fdd4a31052f. Latest consensus state: current_term: 1 leader_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35363 } } }
I20260812 00:11:57.448539 6764 sys_catalog.cc:458] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:57.448840 6762 sys_catalog.cc:455] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add71c01b7fb4ec8aac40fdd4a31052f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35363 } } }
I20260812 00:11:57.449281 6762 sys_catalog.cc:458] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f [sys.catalog]: This master's current role is: LEADER
W20260812 00:11:57.449409 6783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:11:57.449470 6783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:11:57.449536 6780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:57.449677 6780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:57.453567 6780 catalog_manager.cc:1347] Generated new cluster ID: 2f03c55ae7fb4cbea94cc442f0783b2c
I20260812 00:11:57.453626 6780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:11:57.457509 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.445929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.31.227.62:35363)
0812 00:11:57.446117 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:57.446133 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:57.446160 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:11:57.446259 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:57.446265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:57.446327 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:57.446438 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:57.447366 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.449210 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.450167 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.450777 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.450928 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.453655 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:57.453661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:57.453661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:57.453675 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:57.455825 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.456220 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.456224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.456224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.456348 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.456569 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.456570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.456570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.456902 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:57.457075 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:57.457148 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 00:11:57.457297 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260812 00:11:57.472847 6780 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:57.473599 6780 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:57.476619 6780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P add71c01b7fb4ec8aac40fdd4a31052f: Generated new TSK 0
I20260812 00:11:57.476795 6780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:57.504020 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35363
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:57.591266 6792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:57.591500 6792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:57.591538 6792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:57.592694 6792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:57.592762 6792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:57.592804 6792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:57.594342 6792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:57.594607 6792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:57.594841 6792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:57.600165 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.600165 6805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:57.600216 6792 server_base.cc:1034] running on GCE node
W20260812 00:11:57.600238 6804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:57.600543 6792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:57.600806 6792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:57.601953 6792 hybrid_clock.cc:648] HybridClock initialized: now 1786493517601936 us; error 42 us; skew 500 ppm
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:57.604831 6792 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:57.605280 6792 webserver.cc:460] Webserver started at http://127.31.227.1:36631/ using document root <none> and password file <none>
I20260812 00:11:57.605680 6792 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:57.605756 6792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:57.605899 6792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:57.606979 6792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "654d1642872b4bfbbd0e257f17ab3477"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "27533e3963dd2320fa33cf8a1cb220de"
server_key_iv: "2afe2fe7117971037e8fc0d4651a355c"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.607342 6792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "654d1642872b4bfbbd0e257f17ab3477"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "27533e3963dd2320fa33cf8a1cb220de"
server_key_iv: "2afe2fe7117971037e8fc0d4651a355c"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.608816 6792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:57.609586 6820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.609743 6792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:11:57.609844 6792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "654d1642872b4bfbbd0e257f17ab3477"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "27533e3963dd2320fa33cf8a1cb220de"
server_key_iv: "2afe2fe7117971037e8fc0d4651a355c"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.609939 6792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:57.625000 6792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:57.627038 6792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:57.627216 6792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:57.627468 6792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:57.627786 6792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:57.627843 6792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.627889 6792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:57.627930 6792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.634151 6792 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:41229
I20260812 00:11:57.634204 7019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:41229 every 8 connection(s)
I20260812 00:11:57.634516 6792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:57.640127 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6792
I20260812 00:11:57.640247 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:57.640506 32652 external_mini_cluster.cc:1428] Setting key 0d79141349f7090ad019e5a036980af4
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:57.644443 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.635332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:55559 (local address 127.31.227.62:35363)
0812 00:11:57.635421 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:57.635423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:57.635601 (+ 178us) server_negotiation.cc:407] Connection header received
0812 00:11:57.636539 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:57.636540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:57.636556 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:57.636614 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:57.637192 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.637729 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.638783 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.639003 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.639066 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.641602 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:57.641604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:57.641604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:57.641612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:57.642993 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.643419 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.643421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.643421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.643487 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.643745 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.643746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.643747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.643868 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:57.643951 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:57.644233 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 00:11:57.644327 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:11:57.644603 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.635295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:35363 (local address 127.31.227.1:55559)
0812 00:11:57.635465 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:57.635507 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:57.636446 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:57.636670 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:57.636674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:57.636924 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:57.637114 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:57.637118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.637851 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.637852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:57.638674 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:57.638676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.639091 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.639092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:57.639197 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.639689 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:57.639698 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:57.641463 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:57.643097 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:57.643105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:57.643107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:57.643332 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:57.643558 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:57.643559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:57.643559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:57.643674 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:57.643942 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:57.643950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:57.644096 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 00:11:57.644209 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:11:57.644367 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260812 00:11:57.645162 7022 heartbeater.cc:347] Connected to a master server at 127.31.227.62:35363
I20260812 00:11:57.645256 7022 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:57.645462 7022 heartbeater.cc:511] Master 127.31.227.62:35363 requested a full tablet report, sending...
I20260812 00:11:57.646123 6685 ts_manager.cc:194] Registered new tserver with Master: 654d1642872b4bfbbd0e257f17ab3477 (127.31.227.1:41229)
I20260812 00:11:57.646943 6685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:55559
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:57.668205 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35363
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:57.752873 7030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:57.753077 7030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:57.753101 7030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:57.754206 7030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:57.754240 7030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:57.754266 7030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:57.755709 7030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:57.755908 7030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:57.756089 7030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:57.761102 7041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.761130 7038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.761130 7039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:57.761373 7030 server_base.cc:1034] running on GCE node
I20260812 00:11:57.761654 7030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:57.761857 7030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:57.763033 7030 hybrid_clock.cc:648] HybridClock initialized: now 1786493517763027 us; error 35 us; skew 500 ppm
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:57.765662 7030 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:57.766172 7030 webserver.cc:460] Webserver started at http://127.31.227.2:35635/ using document root <none> and password file <none>
I20260812 00:11:57.766435 7030 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:57.766507 7030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:57.766664 7030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:57.767745 7030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "49c2203f04024c4d869d40478ef92b3a"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "0a1c38dfefabccc9275ec62bc49239f4"
server_key_iv: "08296eadbfb473b08c0f6705f6b443a8"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.768127 7030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "49c2203f04024c4d869d40478ef92b3a"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "0a1c38dfefabccc9275ec62bc49239f4"
server_key_iv: "08296eadbfb473b08c0f6705f6b443a8"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.769608 7030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:57.770427 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.770617 7030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:57.770722 7030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "49c2203f04024c4d869d40478ef92b3a"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "0a1c38dfefabccc9275ec62bc49239f4"
server_key_iv: "08296eadbfb473b08c0f6705f6b443a8"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.770812 7030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:57.787900 7030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:57.790325 7030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:57.790498 7030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:57.790733 7030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:57.791057 7030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:57.791114 7030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.791157 7030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:57.791183 7030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.797335 7030 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:45113
I20260812 00:11:57.797420 7260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:45113 every 8 connection(s)
I20260812 00:11:57.797780 7030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:57.803663 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7030
I20260812 00:11:57.803773 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:57.803964 32652 external_mini_cluster.cc:1428] Setting key 203612f5c581e6e30d74ec01eeb813de
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:57.807829 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.798538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46157 (local address 127.31.227.62:35363)
0812 00:11:57.798611 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:57.798613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:57.798842 (+ 229us) server_negotiation.cc:407] Connection header received
0812 00:11:57.799816 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:57.799818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:57.799844 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:57.799913 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:57.800527 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.801048 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.802125 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.802353 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.802434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.804699 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:57.804700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:57.804701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:57.804725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:57.806382 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.806819 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.806820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.806821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.806885 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.807160 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.807161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.807161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.807277 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:57.807352 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:57.807647 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 00:11:57.807742 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:11:57.808017 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.798513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:35363 (local address 127.31.227.2:46157)
0812 00:11:57.798703 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:57.798743 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:57.799696 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:57.799984 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:57.799989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:57.800240 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:57.800428 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:57.800432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.801149 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.801151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:57.802031 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:57.802034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.802443 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.802444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:57.802564 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.802970 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:57.802979 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:57.804623 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:57.806473 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:57.806480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:57.806482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:57.806711 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:57.806966 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:57.806967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:57.806968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:57.807080 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:57.807344 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:57.807352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:57.807503 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 00:11:57.807611 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:11:57.807767 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260812 00:11:57.808576 7261 heartbeater.cc:347] Connected to a master server at 127.31.227.62:35363
I20260812 00:11:57.808660 7261 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:57.808847 7261 heartbeater.cc:511] Master 127.31.227.62:35363 requested a full tablet report, sending...
I20260812 00:11:57.809300 6685 ts_manager.cc:194] Registered new tserver with Master: 49c2203f04024c4d869d40478ef92b3a (127.31.227.2:45113)
I20260812 00:11:57.809890 6685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:46157
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:57.832355 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35363
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:57.917232 7275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:57.917474 7275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:57.917517 7275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:57.918872 7275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:57.918941 7275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:57.918985 7275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:11:57.921157 7275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:57.921452 7275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:57.921723 7275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:57.926774 7283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.926824 7282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:57.926795 7287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:57.927038 7275 server_base.cc:1034] running on GCE node
I20260812 00:11:57.927238 7275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:57.927471 7275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:57.928611 7275 hybrid_clock.cc:648] HybridClock initialized: now 1786493517928601 us; error 35 us; skew 500 ppm
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:57.931197 7275 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:11:57.931707 7275 webserver.cc:460] Webserver started at http://127.31.227.3:34795/ using document root <none> and password file <none>
I20260812 00:11:57.931944 7275 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:57.931990 7275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:57.932163 7275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:57.933245 7275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "368a908c9c654da999eed223e55307c8"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "e007d8908c6a77455c30cc8d5e350e55"
server_key_iv: "431dfc85491c132986aec9e7540ab25e"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.933653 7275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "368a908c9c654da999eed223e55307c8"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "e007d8908c6a77455c30cc8d5e350e55"
server_key_iv: "431dfc85491c132986aec9e7540ab25e"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.935173 7275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:57.936110 7299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.936296 7275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:57.936426 7275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "368a908c9c654da999eed223e55307c8"
format_stamp: "Formatted at 2026-08-12 00:11:57 on dist-test-slave-zr1t"
server_key: "e007d8908c6a77455c30cc8d5e350e55"
server_key_iv: "431dfc85491c132986aec9e7540ab25e"
server_key_version: "encryptionkey@0"
I20260812 00:11:57.936509 7275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:57.972771 7275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:57.975103 7275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:57.975279 7275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:57.975548 7275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:57.975888 7275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:57.975943 7275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.975991 7275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:57.976014 7275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:57.983419 7275 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:42265
I20260812 00:11:57.983464 7494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:42265 every 8 connection(s)
I20260812 00:11:57.983819 7275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:11:57.988735 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7275
I20260812 00:11:57.988821 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:57.989027 32652 external_mini_cluster.cc:1428] Setting key ca2df2baa6405d6f761ae6a7741f247f
Aug 12 00:11:57 dist-test-slave-zr1t krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:57.993384 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.984614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58289 (local address 127.31.227.62:35363)
0812 00:11:57.984681 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:57.984683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:57.984888 (+ 205us) server_negotiation.cc:407] Connection header received
0812 00:11:57.985927 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:57.985928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:57.985939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:57.986005 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:57.986599 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.987090 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.988069 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.988246 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.988294 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.990913 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:57.990915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:57.990916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:57.990922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:57.992154 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.992567 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.992568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.992568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.992622 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:57.992850 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:57.992850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:57.992850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:57.992959 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:57.993013 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:57.993252 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 00:11:57.993318 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:11:57.993640 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:57.984563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:35363 (local address 127.31.227.3:58289)
0812 00:11:57.984744 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:57.984788 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:57.985832 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:57.986066 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:57.986069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:57.986323 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:57.986527 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:57.986531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.987172 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.987174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:57.987996 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:57.987999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:57.988318 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:57.988319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:57.988438 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:57.989123 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:57.989135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:57.990818 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:57.992253 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:57.992260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:57.992262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:57.992490 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:57.992688 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:57.992689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:57.992689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:57.992792 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:57.993017 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:57.993022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:57.993139 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 00:11:57.993222 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:11:57.993344 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260812 00:11:57.994184 7498 heartbeater.cc:347] Connected to a master server at 127.31.227.62:35363
I20260812 00:11:57.994305 7498 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:57.994498 7498 heartbeater.cc:511] Master 127.31.227.62:35363 requested a full tablet report, sending...
I20260812 00:11:57.994853 6685 ts_manager.cc:194] Registered new tserver with Master: 368a908c9c654da999eed223e55307c8 (127.31.227.3:42265)
I20260812 00:11:57.995215 6685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:58289
I20260812 00:11:58.000910 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:11:58 dist-test-slave-zr1t krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:11:58 dist-test-slave-zr1t krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:11:58.025468 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:58.017417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.31.227.1:41229)
0812 00:11:58.017637 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:58.017642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:58.017652 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:58.017680 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:58.017682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:58.017697 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:58.017767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:58.018459 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:58.019234 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:58.019942 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:58.020253 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:58.020308 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:58.022554 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:58.022564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:58.022564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:58.022578 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:58.024219 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:58.024599 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:58.024603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:58.024603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:58.024663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:58.024886 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:58.024887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:58.024888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:58.025029 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:58.025198 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:58.025258 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 00:11:58.025336 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":135,"threads_started":1}
Aug 12 00:11:58 dist-test-slave-zr1t krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:58.033072 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:58.026830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.31.227.62:35363)
0812 00:11:58.026875 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:58.026877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:58.026888 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:11:58.026934 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:58.026935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:58.026943 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:58.026987 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:58.027375 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:58.027831 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:58.028462 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:58.028638 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:58.028686 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:58.030691 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:58.030692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:58.030693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:58.030699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:58.032096 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:58.032432 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:58.032433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:58.032433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:58.032491 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:58.032653 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:58.032653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:58.032653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:58.032779 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:58.032845 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:58.032956 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 00:11:58.033013 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 00:11:58.033313 6685 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39234
I20260812 00:11:58.033690 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6792
I20260812 00:11:58.046733 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7030
I20260812 00:11:58.067610 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7275
I20260812 00:11:58.076730 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6627
2026-08-12T00:11:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2188 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:58 dist-test-slave-zr1t krb5kdc[7524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:58 dist-test-slave-zr1t krb5kdc[7524](info): set up 2 sockets
Aug 12 00:11:58 dist-test-slave-zr1t krb5kdc[7524](info): commencing operation
krb5kdc: starting...
W20260812 00:11:59.354194 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:11:59.477742 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42707
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:59.560931 7551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:59.561163 7551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:59.561201 7551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:59.562408 7551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:59.562479 7551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:59.562503 7551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:59.562532 7551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:59.562563 7551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:59.564033 7551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:59.564306 7551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:59.564543 7551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:11:59.569420 7551 server_base.cc:1034] running on GCE node
W20260812 00:11:59.569330 7562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:59.569330 7561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:59.569346 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:59.569851 7551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:59.570148 7551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:59.571321 7551 hybrid_clock.cc:648] HybridClock initialized: now 1786493519571303 us; error 27 us; skew 500 ppm
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:59.573908 7551 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:11:59.574203 7551 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:59.574245 7551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:59.574326 7551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:59.575300 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "f51fc33b821c45d592c83ed5ab509d7a"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "f0758b90f56e3d57552957a3331521b8"
server_key_iv: "142b6115493c2ab2156bbedc22b65017"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.575603 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "f51fc33b821c45d592c83ed5ab509d7a"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "f0758b90f56e3d57552957a3331521b8"
server_key_iv: "142b6115493c2ab2156bbedc22b65017"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.576781 7551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:11:59.577425 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.577620 7551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.577723 7551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "f51fc33b821c45d592c83ed5ab509d7a"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "f0758b90f56e3d57552957a3331521b8"
server_key_iv: "142b6115493c2ab2156bbedc22b65017"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.577793 7551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:59.588349 7551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:59.590171 7551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:59.590322 7551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:59.594455 7551 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42707
I20260812 00:11:59.594478 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42707 every 8 connection(s)
I20260812 00:11:59.594753 7551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:59.595772 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:59.598073 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: Bootstrap starting.
I20260812 00:11:59.598644 7671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:59.598903 7671 log.cc:824] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:59.599602 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: No bootstrap required, opened a new log
I20260812 00:11:59.601207 7671 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51fc33b821c45d592c83ed5ab509d7a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42707 } }
I20260812 00:11:59.601346 7671 raft_consensus.cc:374] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:59.601368 7671 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51fc33b821c45d592c83ed5ab509d7a, State: Initialized, Role: FOLLOWER
I20260812 00:11:59.601481 7671 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51fc33b821c45d592c83ed5ab509d7a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42707 } }
I20260812 00:11:59.601605 7671 raft_consensus.cc:388] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:59.601631 7671 raft_consensus.cc:482] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:59.601660 7671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:59.602490 7671 raft_consensus.cc:504] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51fc33b821c45d592c83ed5ab509d7a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42707 } }
I20260812 00:11:59.602582 7671 leader_election.cc:302] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f51fc33b821c45d592c83ed5ab509d7a; no voters:
I20260812 00:11:59.602725 7671 leader_election.cc:288] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:59.602787 7680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:59.603047 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7551
I20260812 00:11:59.603159 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:11:59.603372 32652 external_mini_cluster.cc:1428] Setting key da5fa1badf44177d7f037d89193f0b92
I20260812 00:11:59.603039 7671 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:59.603602 7680 raft_consensus.cc:686] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [term 1 LEADER]: Becoming Leader. State: Replica: f51fc33b821c45d592c83ed5ab509d7a, State: Running, Role: LEADER
I20260812 00:11:59.603710 7680 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51fc33b821c45d592c83ed5ab509d7a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42707 } }
I20260812 00:11:59.604390 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51fc33b821c45d592c83ed5ab509d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51fc33b821c45d592c83ed5ab509d7a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42707 } } }
I20260812 00:11:59.604475 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:59.604799 7687 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51fc33b821c45d592c83ed5ab509d7a. Latest consensus state: current_term: 1 leader_uuid: "f51fc33b821c45d592c83ed5ab509d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51fc33b821c45d592c83ed5ab509d7a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42707 } } }
I20260812 00:11:59.604967 7687 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a [sys.catalog]: This master's current role is: LEADER
W20260812 00:11:59.605445 7714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:11:59.605515 7714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:11:59.605671 7706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:59.605850 7706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:11:59.608179 7706 catalog_manager.cc:1347] Generated new cluster ID: ccca74bd5b7e4e70a86b5ac67ac99cc5
I20260812 00:11:59.608237 7706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:59.615540 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:59.604012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.31.227.62:42707)
0812 00:11:59.604255 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:59.604264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:59.604298 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:11:59.604435 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:59.604442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:59.604502 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:59.605519 (+ 1017us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:59.605655 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.607045 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.607856 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.608510 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.608635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:59.611568 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:59.611572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:59.611573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:59.611585 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:59.613828 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:59.614252 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:59.614256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:59.614257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:59.614365 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:59.614589 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:59.614590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:59.614591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:59.614891 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:59.615049 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:59.615185 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 00:11:59.615324 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":117,"threads_started":1}
I20260812 00:11:59.620100 7706 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:59.620862 7706 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:59.626230 7706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: Generated new TSK 0
I20260812 00:11:59.626430 7706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:11:59.652259 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42707
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:59.738808 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:59.739043 7725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:59.739086 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:59.740269 7725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:59.740332 7725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:59.740377 7725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:11:59.742060 7725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:59.742336 7725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:59.742573 7725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:59.747697 7740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:59.747701 7744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:59.747757 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:59.747934 7725 server_base.cc:1034] running on GCE node
I20260812 00:11:59.748162 7725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:59.748445 7725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:59.749611 7725 hybrid_clock.cc:648] HybridClock initialized: now 1786493519749587 us; error 38 us; skew 500 ppm
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:59.752310 7725 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:11:59.752878 7725 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:59.752967 7725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:59.753124 7725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:59.754415 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "291a2a33fea54065a063d3dcdabee97f"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "53b04c9078fe61e0fc8b95222aa2fac6"
server_key_iv: "8533a99ad3df38b7c2c362261410fe15"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.754841 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "291a2a33fea54065a063d3dcdabee97f"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "53b04c9078fe61e0fc8b95222aa2fac6"
server_key_iv: "8533a99ad3df38b7c2c362261410fe15"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.756347 7725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:11:59.757227 7759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.757428 7725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:59.757562 7725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "291a2a33fea54065a063d3dcdabee97f"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "53b04c9078fe61e0fc8b95222aa2fac6"
server_key_iv: "8533a99ad3df38b7c2c362261410fe15"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.757660 7725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:59.777413 7725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:59.779345 7725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:59.779505 7725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:59.779742 7725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:59.780072 7725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:59.780130 7725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.780164 7725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:59.780189 7725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.786566 7725 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:42467
I20260812 00:11:59.786600 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:42467 every 8 connection(s)
I20260812 00:11:59.786877 7725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:59.788305 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7725
I20260812 00:11:59.788417 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:59.788601 32652 external_mini_cluster.cc:1428] Setting key 799a66ba52d44bcad6a1bf080088d0ec
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:11:59.797610 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:59.787603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:56485 (local address 127.31.227.62:42707)
0812 00:11:59.787666 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:59.787668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:59.787871 (+ 203us) server_negotiation.cc:407] Connection header received
0812 00:11:59.788478 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:59.788479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:59.788492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:59.788528 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:59.789788 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.790599 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.791717 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.791942 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.792032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:59.794559 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:59.794560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:59.794561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:59.794568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:59.795992 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:59.796506 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:59.796506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:59.796507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:59.796559 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:59.796880 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:59.796880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:59.796881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:59.796978 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:59.797026 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:59.797405 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 00:11:59.797482 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:11:59.797819 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:59.787589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42707 (local address 127.31.227.1:56485)
0812 00:11:59.787747 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:59.787782 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:59.788335 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:59.788710 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:59.788714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:59.789062 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:59.789695 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:59.789701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.790727 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.790729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:59.791603 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:59.791606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.792048 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.792049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:59.792180 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:59.792613 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:59.792621 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:59.794439 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:59.796092 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:59.796101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:59.796103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:59.796422 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:59.796626 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:59.796627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:59.796627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:59.796809 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:59.797045 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:59.797053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:59.797216 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 00:11:59.797351 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 00:11:59.797547 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260812 00:11:59.798378 7960 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42707
I20260812 00:11:59.798492 7960 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
I20260812 00:11:59.798671 7960 heartbeater.cc:511] Master 127.31.227.62:42707 requested a full tablet report, sending...
I20260812 00:11:59.799208 7611 ts_manager.cc:194] Registered new tserver with Master: 291a2a33fea54065a063d3dcdabee97f (127.31.227.1:42467)
I20260812 00:11:59.799932 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:56485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:11:59.814472 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42707
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:59.898061 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:59.898260 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:59.898284 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:59.899372 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:59.899407 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:59.899433 7971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:11:59.900923 7971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:11:59.901117 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:59.901295 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:59.906524 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:59.906587 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:59.906504 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:59.906724 7971 server_base.cc:1034] running on GCE node
I20260812 00:11:59.906967 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:59.907262 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:59.908442 7971 hybrid_clock.cc:648] HybridClock initialized: now 1786493519908420 us; error 35 us; skew 500 ppm
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:59.911108 7971 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:11:59.911494 7971 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:59.911545 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:59.911638 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:11:59.912672 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "460fbb7202124dc89db2baa4df0f2c0d"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "4f9cf8f515722d29883f6487110b299f"
server_key_iv: "304cb5a3c328c0c149b3e03f25779e4f"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.913070 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "460fbb7202124dc89db2baa4df0f2c0d"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "4f9cf8f515722d29883f6487110b299f"
server_key_iv: "304cb5a3c328c0c149b3e03f25779e4f"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.914599 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:11:59.915405 8000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.915639 7971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:11:59.915719 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "460fbb7202124dc89db2baa4df0f2c0d"
format_stamp: "Formatted at 2026-08-12 00:11:59 on dist-test-slave-zr1t"
server_key: "4f9cf8f515722d29883f6487110b299f"
server_key_iv: "304cb5a3c328c0c149b3e03f25779e4f"
server_key_version: "encryptionkey@0"
I20260812 00:11:59.915807 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:59.929189 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:59.931383 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:59.931542 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:59.931803 7971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:59.932264 7971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:59.932328 7971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.932358 7971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:59.932371 7971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:59.938815 7971 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:36545
I20260812 00:11:59.938866 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:36545 every 8 connection(s)
I20260812 00:11:59.939146 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:59.939350 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7971
I20260812 00:11:59.939477 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:59.939720 32652 external_mini_cluster.cc:1428] Setting key 65b6d2df3f580703a2154ead3b2103b5
Aug 12 00:11:59 dist-test-slave-zr1t krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493519, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
I20260812 00:11:59.950225 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:59.939943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:55751 (local address 127.31.227.62:42707)
0812 00:11:59.940020 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:59.940023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:59.940282 (+ 259us) server_negotiation.cc:407] Connection header received
0812 00:11:59.940937 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:59.940937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:59.940961 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:59.941006 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:59.942414 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.942913 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.944370 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.944561 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.944654 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:59.947193 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:59.947195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:59.947195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:59.947204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:59.948690 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:59.949136 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:59.949137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:59.949138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:59.949201 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:59.949480 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:59.949481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:59.949482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:59.949647 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:59.949729 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:59.950037 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 00:11:59.950126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:11:59.950434 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:59.939932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42707 (local address 127.31.227.2:55751)
0812 00:11:59.940154 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:59.940207 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:59.940828 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:59.941084 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:59.941088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:59.941573 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:59.942319 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:59.942328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.943014 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.943016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:59.944288 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:59.944291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:59.944667 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:59.944668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:59.944797 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:59.945361 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:59.945370 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:59.947085 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:59.948800 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:59.948809 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:59.948812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:59.949053 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:59.949272 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:59.949273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:59.949273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:59.949392 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:59.949721 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:59.949729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:59.949899 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 00:11:59.950017 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:11:59.950192 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":115,"threads_started":1}
I20260812 00:11:59.950955 8206 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42707
I20260812 00:11:59.951063 8206 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:59.951195 8206 heartbeater.cc:511] Master 127.31.227.62:42707 requested a full tablet report, sending...
I20260812 00:11:59.951586 7611 ts_manager.cc:194] Registered new tserver with Master: 460fbb7202124dc89db2baa4df0f2c0d (127.31.227.2:36545)
I20260812 00:11:59.951985 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:55751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:11:59.966593 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42707
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:00.056281 8211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:00.056551 8211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:00.056593 8211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:00.057891 8211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:00.057953 8211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:00.057998 8211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:00.060519 8211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:00.060791 8211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:00.061049 8211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:00.069322 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.069386 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.069300 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.069900 8211 server_base.cc:1034] running on GCE node
I20260812 00:12:00.070135 8211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:00.070452 8211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:00.071647 8211 hybrid_clock.cc:648] HybridClock initialized: now 1786493520071634 us; error 28 us; skew 500 ppm
Aug 12 00:12:00 dist-test-slave-zr1t krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493520, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:00.075301 8211 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:12:00.075706 8211 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:00.075781 8211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:00.075932 8211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:00.077226 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "930141660dae46b0ad413bd0e5582064"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "2e3c37942a24b64d479783c936b9d77a"
server_key_iv: "2d99f690cd7f075a626a089b15622968"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.077696 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "930141660dae46b0ad413bd0e5582064"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "2e3c37942a24b64d479783c936b9d77a"
server_key_iv: "2d99f690cd7f075a626a089b15622968"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.079001 8211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:00.079736 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.079948 8211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:00.080063 8211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "930141660dae46b0ad413bd0e5582064"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "2e3c37942a24b64d479783c936b9d77a"
server_key_iv: "2d99f690cd7f075a626a089b15622968"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.080156 8211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:00.091949 8211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:00.094260 8211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:00.094414 8211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:00.094641 8211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:00.094951 8211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:00.095007 8211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.095057 8211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:00.095084 8211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.101306 8211 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41297
I20260812 00:12:00.101346 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41297 every 8 connection(s)
I20260812 00:12:00.101709 8211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:00.105299 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 8211
I20260812 00:12:00.105422 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:00.105664 32652 external_mini_cluster.cc:1428] Setting key 04161dbe000e9c676dbda9e31c93fd50
Aug 12 00:12:00 dist-test-slave-zr1t krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493520, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:00.111701 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.102407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:47203 (local address 127.31.227.62:42707)
0812 00:12:00.102463 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.102465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.102690 (+ 225us) server_negotiation.cc:407] Connection header received
0812 00:12:00.103316 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.103318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.103331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.103391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.104524 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.105071 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.106523 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.106722 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.106812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.109053 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.109055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.109055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:00.109063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.110302 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:00.110707 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:00.110708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:00.110708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:00.110769 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:00.111037 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:00.111038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:00.111039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:00.111153 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.111224 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.111521 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.111611 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:12:00.111886 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.102384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42707 (local address 127.31.227.3:47203)
0812 00:12:00.102564 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.102602 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.103208 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.103462 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.103466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.103735 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:00.104420 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.104426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.105204 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.105206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.106438 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.106441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.106838 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.106839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.106978 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.107540 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:00.107549 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:00.108951 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:00.110389 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:00.110397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:00.110400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:00.110627 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:00.110843 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:00.110844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:00.110844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:00.110961 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:00.111229 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:00.111237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:00.111381 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.111486 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.111643 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260812 00:12:00.112501 8417 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42707
I20260812 00:12:00.112600 8417 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:00.112778 8417 heartbeater.cc:511] Master 127.31.227.62:42707 requested a full tablet report, sending...
I20260812 00:12:00.113322 7611 ts_manager.cc:194] Registered new tserver with Master: 930141660dae46b0ad413bd0e5582064 (127.31.227.3:41297)
I20260812 00:12:00.113760 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:47203
I20260812 00:12:00.117442 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:00.124886 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.118759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.31.227.62:42707)
0812 00:12:00.118824 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.118826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.119005 (+ 179us) server_negotiation.cc:407] Connection header received
0812 00:12:00.119044 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.119044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.119053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.119100 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.119617 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.120150 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.120869 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.121054 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.121112 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.122325 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.122326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.122327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:00.122332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.123680 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:00.123996 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:00.123997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:00.123998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:00.124049 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:00.124214 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:00.124214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:00.124215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:00.124381 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.124462 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.124659 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.124732 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:12:00.126868 7611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:00.128461 7611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:00.135553 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.131134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.31.227.1:42467)
0812 00:12:00.131906 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.131911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.131920 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:00.131957 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.131959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.131976 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.132064 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:00.132206 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.133094 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.134437 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.135120 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.135187 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.135237 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.135414 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.135459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":112,"threads_started":1}
I20260812 00:12:00.135553 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.130979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:36545 (local address 127.0.0.1:43140)
0812 00:12:00.131323 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.131381 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.131456 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.131749 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.131752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.131755 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:00.131834 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.131837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.133098 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.133099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.134195 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.134198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.135258 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.135259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.135348 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.135351 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.135425 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.135468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260812 00:12:00.135561 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.130981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:42467 (local address 127.0.0.1:52434)
0812 00:12:00.131349 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.131384 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.131449 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.132082 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.132083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.132085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:00.132128 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.132129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.133201 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.133202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.134347 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.134349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.135205 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.135206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.135288 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.135302 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.135394 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.135456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":89,"threads_started":1}
I20260812 00:12:00.136019 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.131094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.31.227.2:36545)
0812 00:12:00.131574 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.131578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.131588 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:00.131634 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.131636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.131649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.131742 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:00.131956 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.132986 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.134298 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.135066 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.135146 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.135217 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.135777 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.135825 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":105,"threads_started":1}
I20260812 00:12:00.136363 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.131197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41297 (local address 127.0.0.1:35486)
0812 00:12:00.132206 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.132217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.132277 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.132468 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.132469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.132471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:00.132510 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.132511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.133772 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.133774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.134711 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.134712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.135524 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.135525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.135584 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.135587 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.136251 (+ 664us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.136294 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":75,"threads_started":1}
I20260812 00:12:00.136377 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.131330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.31.227.3:41297)
0812 00:12:00.131542 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.131547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.132296 (+ 749us) server_negotiation.cc:407] Connection header received
0812 00:12:00.132344 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.132346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.132373 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.132454 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:00.132572 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.133662 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.134798 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.135373 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.135433 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.135487 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.136270 (+ 783us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.136311 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":106,"threads_started":1}
I20260812 00:12:00.136441 7854 tablet_service.cc:1467] Processing CreateTablet for tablet fca071c391914b6fa53b82b76a4a424c (DEFAULT_TABLE table=test-table [id=83e0e55b2e5c456f9dfc3ff79aa58f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:00.136852 7854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca071c391914b6fa53b82b76a4a424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:00.136998 8316 tablet_service.cc:1467] Processing CreateTablet for tablet fca071c391914b6fa53b82b76a4a424c (DEFAULT_TABLE table=test-table [id=83e0e55b2e5c456f9dfc3ff79aa58f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:00.137336 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca071c391914b6fa53b82b76a4a424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:00.138162 8086 tablet_service.cc:1467] Processing CreateTablet for tablet fca071c391914b6fa53b82b76a4a424c (DEFAULT_TABLE table=test-table [id=83e0e55b2e5c456f9dfc3ff79aa58f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:00.138511 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca071c391914b6fa53b82b76a4a424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:00.139916 8445 tablet_bootstrap.cc:492] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: Bootstrap starting.
I20260812 00:12:00.140636 8445 tablet_bootstrap.cc:654] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:00.140940 8447 tablet_bootstrap.cc:492] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Bootstrap starting.
I20260812 00:12:00.141551 8445 log.cc:824] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:00.141733 8447 tablet_bootstrap.cc:654] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:00.141836 8448 tablet_bootstrap.cc:492] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Bootstrap starting.
I20260812 00:12:00.142383 8447 log.cc:824] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:00.142486 8448 tablet_bootstrap.cc:654] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:00.142656 8445 tablet_bootstrap.cc:492] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: No bootstrap required, opened a new log
I20260812 00:12:00.142742 8445 ts_tablet_manager.cc:1397] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:00.143191 8447 tablet_bootstrap.cc:492] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: No bootstrap required, opened a new log
I20260812 00:12:00.143275 8447 ts_tablet_manager.cc:1397] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:00.143308 8448 log.cc:824] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:00.144114 8448 tablet_bootstrap.cc:492] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: No bootstrap required, opened a new log
I20260812 00:12:00.144182 8448 ts_tablet_manager.cc:1397] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:00.144992 8445 raft_consensus.cc:348] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.145169 8445 raft_consensus.cc:374] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:00.145208 8445 raft_consensus.cc:729] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291a2a33fea54065a063d3dcdabee97f, State: Initialized, Role: FOLLOWER
I20260812 00:12:00.145135 8447 raft_consensus.cc:348] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.145316 8447 raft_consensus.cc:374] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:00.145371 8447 raft_consensus.cc:729] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930141660dae46b0ad413bd0e5582064, State: Initialized, Role: FOLLOWER
I20260812 00:12:00.145345 8445 consensus_queue.cc:260] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.145493 8447 consensus_queue.cc:260] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.145759 8445 ts_tablet_manager.cc:1428] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:00.145838 8447 ts_tablet_manager.cc:1428] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:00.145848 7960 heartbeater.cc:503] Master 127.31.227.62:42707 was elected leader, sending a full tablet report...
I20260812 00:12:00.145964 8417 heartbeater.cc:503] Master 127.31.227.62:42707 was elected leader, sending a full tablet report...
I20260812 00:12:00.146682 8448 raft_consensus.cc:348] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.146865 8448 raft_consensus.cc:374] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:00.146919 8448 raft_consensus.cc:729] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460fbb7202124dc89db2baa4df0f2c0d, State: Initialized, Role: FOLLOWER
I20260812 00:12:00.147046 8448 consensus_queue.cc:260] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.147347 8448 ts_tablet_manager.cc:1428] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:00.147516 8206 heartbeater.cc:503] Master 127.31.227.62:42707 was elected leader, sending a full tablet report...
W20260812 00:12:00.189973 8207 tablet.cc:2369] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:00.287586 7961 tablet.cc:2369] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:00.352320 8419 tablet.cc:2369] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:00.352356 8454 raft_consensus.cc:482] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:00.352528 8454 raft_consensus.cc:504] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.352967 8454 leader_election.cc:288] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 460fbb7202124dc89db2baa4df0f2c0d (127.31.227.2:36545), 291a2a33fea54065a063d3dcdabee97f (127.31.227.1:42467)
I20260812 00:12:00.357084 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.353227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:48285 (local address 127.31.227.2:36545)
0812 00:12:00.353276 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.353278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.353308 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:12:00.353397 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.353398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.353430 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.353510 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:00.353750 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.354516 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.356131 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.356699 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.356781 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.356790 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.356965 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.357007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:00.357084 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.353248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:53697 (local address 127.31.227.1:42467)
0812 00:12:00.353297 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.353300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.353474 (+ 174us) server_negotiation.cc:407] Connection header received
0812 00:12:00.353550 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.353551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.353563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.353624 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:00.353763 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.354511 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.356132 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.356699 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.356781 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.356790 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.356971 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.357009 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:12:00.357124 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.353157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:36545 (local address 127.31.227.3:48285)
0812 00:12:00.353248 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.353261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.353322 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.353561 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.353563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.353565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:00.353645 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.353647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.354609 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.354610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.356025 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.356029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.356790 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.356791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.356882 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.356884 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.356938 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.356995 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":83}
I20260812 00:12:00.357133 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.353180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:42467 (local address 127.31.227.3:53697)
0812 00:12:00.353393 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.353402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.353460 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.353634 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.353635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.353636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:00.353692 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.353693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.354609 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.354610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.356025 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:00.356029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.356790 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.356791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:00.356873 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.356877 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.356942 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.356995 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":74,"thread_start_us":141,"threads_started":1}
I20260812 00:12:00.357589 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca071c391914b6fa53b82b76a4a424c" candidate_uuid: "930141660dae46b0ad413bd0e5582064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460fbb7202124dc89db2baa4df0f2c0d" is_pre_election: true
I20260812 00:12:00.357695 7883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca071c391914b6fa53b82b76a4a424c" candidate_uuid: "930141660dae46b0ad413bd0e5582064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "291a2a33fea54065a063d3dcdabee97f" is_pre_election: true
I20260812 00:12:00.357759 8120 raft_consensus.cc:2454] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930141660dae46b0ad413bd0e5582064 in term 0.
I20260812 00:12:00.357861 7883 raft_consensus.cc:2454] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930141660dae46b0ad413bd0e5582064 in term 0.
I20260812 00:12:00.358033 8237 leader_election.cc:302] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 460fbb7202124dc89db2baa4df0f2c0d, 930141660dae46b0ad413bd0e5582064; no voters:
I20260812 00:12:00.358218 8454 raft_consensus.cc:2781] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:00.358258 8454 raft_consensus.cc:482] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:00.358275 8454 raft_consensus.cc:3037] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:00.358997 8454 raft_consensus.cc:504] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.359148 8454 leader_election.cc:288] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [CANDIDATE]: Term 1 election: Requested vote from peers 460fbb7202124dc89db2baa4df0f2c0d (127.31.227.2:36545), 291a2a33fea54065a063d3dcdabee97f (127.31.227.1:42467)
I20260812 00:12:00.359297 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca071c391914b6fa53b82b76a4a424c" candidate_uuid: "930141660dae46b0ad413bd0e5582064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460fbb7202124dc89db2baa4df0f2c0d"
I20260812 00:12:00.359377 8120 raft_consensus.cc:3037] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:00.359339 7883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca071c391914b6fa53b82b76a4a424c" candidate_uuid: "930141660dae46b0ad413bd0e5582064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "291a2a33fea54065a063d3dcdabee97f"
I20260812 00:12:00.359442 7883 raft_consensus.cc:3037] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:00.360105 8120 raft_consensus.cc:2454] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930141660dae46b0ad413bd0e5582064 in term 1.
I20260812 00:12:00.360229 7883 raft_consensus.cc:2454] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930141660dae46b0ad413bd0e5582064 in term 1.
I20260812 00:12:00.360292 8237 leader_election.cc:302] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 460fbb7202124dc89db2baa4df0f2c0d, 930141660dae46b0ad413bd0e5582064; no voters:
I20260812 00:12:00.360385 8454 raft_consensus.cc:2781] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:00.360589 8454 raft_consensus.cc:686] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 1 LEADER]: Becoming Leader. State: Replica: 930141660dae46b0ad413bd0e5582064, State: Running, Role: LEADER
I20260812 00:12:00.360716 8454 consensus_queue.cc:237] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } }
I20260812 00:12:00.361709 7611 catalog_manager.cc:5495] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 930141660dae46b0ad413bd0e5582064 (127.31.227.3). New cstate: current_term: 1 leader_uuid: "930141660dae46b0ad413bd0e5582064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "930141660dae46b0ad413bd0e5582064" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41297 } health_report { overall_health: HEALTHY } } }
I20260812 00:12:00.367420 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.364890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.31.227.3:41297)
0812 00:12:00.364957 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.364958 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.364966 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:00.365006 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.365007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.365024 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.365099 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:00.365267 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.365826 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.366594 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:00.366804 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:00.366882 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:00.367050 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:00.367161 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:00.367251 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.367291 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.367312 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:12:00.369339 7883 raft_consensus.cc:1261] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 1 FOLLOWER]: Refusing update from remote peer 930141660dae46b0ad413bd0e5582064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:00.369330 8120 raft_consensus.cc:1261] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 1 FOLLOWER]: Refusing update from remote peer 930141660dae46b0ad413bd0e5582064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:00.369751 8454 consensus_queue.cc:1035] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a2a33fea54065a063d3dcdabee97f" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:00.369896 8458 consensus_queue.cc:1035] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "460fbb7202124dc89db2baa4df0f2c0d" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:00.373030 8463 mvcc.cc:205] Tried to move back new op lower bound from 7317477459429584896 to 7317477459398168576. Current Snapshot: MvccSnapshot[applied={T|T < 7317477459429584896}]
I20260812 00:12:00.373106 8462 mvcc.cc:205] Tried to move back new op lower bound from 7317477459429584896 to 7317477459398168576. Current Snapshot: MvccSnapshot[applied={T|T < 7317477459429584896 or (T in {7317477459429584896})}]
I20260812 00:12:00.374938 7611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:00.375046 7611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:00.375838 7611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f51fc33b821c45d592c83ed5ab509d7a: Sending DeleteTablet for 3 replicas of tablet fca071c391914b6fa53b82b76a4a424c
I20260812 00:12:00.376204 7854 tablet_service.cc:1514] Processing DeleteTablet for tablet fca071c391914b6fa53b82b76a4a424c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:00 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:52434
I20260812 00:12:00.376286 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet fca071c391914b6fa53b82b76a4a424c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:00 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:35486
I20260812 00:12:00.376292 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet fca071c391914b6fa53b82b76a4a424c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:00 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:43140
I20260812 00:12:00.376659 8476 tablet_replica.cc:330] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f: stopping tablet replica
I20260812 00:12:00.376731 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7725
I20260812 00:12:00.376770 8476 raft_consensus.cc:2229] T fca071c391914b6fa53b82b76a4a424c P 291a2a33fea54065a063d3dcdabee97f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:00.404251 8477 tablet_replica.cc:330] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: stopping tablet replica
I20260812 00:12:00.404470 8477 raft_consensus.cc:2229] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:00.404650 8477 raft_consensus.cc:2258] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:00.407454 8477 ts_tablet_manager.cc:1905] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:00.410069 8477 ts_tablet_manager.cc:1918] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:00.410174 8477 log.cc:1196] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/fca071c391914b6fa53b82b76a4a424c
I20260812 00:12:00.410537 8477 ts_tablet_manager.cc:1939] T fca071c391914b6fa53b82b76a4a424c P 930141660dae46b0ad413bd0e5582064: Deleting consensus metadata
I20260812 00:12:00.411525 8481 tablet_replica.cc:330] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: stopping tablet replica
I20260812 00:12:00.411700 8481 raft_consensus.cc:2229] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:00.411873 8481 raft_consensus.cc:2258] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:00.412864 8481 ts_tablet_manager.cc:1905] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:00.413028 7581 catalog_manager.cc:4841] TS 930141660dae46b0ad413bd0e5582064 (127.31.227.3:41297): tablet fca071c391914b6fa53b82b76a4a424c (table test-table [id=83e0e55b2e5c456f9dfc3ff79aa58f14]) successfully deleted
I20260812 00:12:00.415073 8481 ts_tablet_manager.cc:1918] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:00.415187 8481 log.cc:1196] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/fca071c391914b6fa53b82b76a4a424c
I20260812 00:12:00.415525 8481 ts_tablet_manager.cc:1939] T fca071c391914b6fa53b82b76a4a424c P 460fbb7202124dc89db2baa4df0f2c0d: Deleting consensus metadata
I20260812 00:12:00.416023 7581 catalog_manager.cc:4841] TS 460fbb7202124dc89db2baa4df0f2c0d (127.31.227.2:36545): tablet fca071c391914b6fa53b82b76a4a424c (table test-table [id=83e0e55b2e5c456f9dfc3ff79aa58f14]) successfully deleted
W20260812 00:12:00.420698 7582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:42467 (error 108)
I20260812 00:12:00.420980 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7971
I20260812 00:12:00.430960 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.430761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:42467 (local address 127.0.0.1:43010)
0812 00:12:00.430833 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.430900 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 00:12:00.431130 7582 catalog_manager.cc:4584] TS 291a2a33fea54065a063d3dcdabee97f (127.31.227.1:42467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fca071c391914b6fa53b82b76a4a424c: Network error: Client connection negotiation failed: client connection to 127.31.227.1:42467: connect: Connection refused (error 111)
I20260812 00:12:00.442724 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 8211
I20260812 00:12:00.454751 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7551
2026-08-12T00:12:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2384 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T00:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:00Z Disabled control of system clock
I20260812 00:12:00.494901 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40117
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:12:00.577350 8488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:00.577575 8488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:00.577597 8488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:00.578661 8488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:00.578696 8488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:00.578709 8488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:00.578722 8488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:00.578733 8488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:00.580005 8488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:40117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:00.580204 8488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:00.580482 8488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:00.585727 8502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.585731 8499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.585847 8488 server_base.cc:1034] running on GCE node
W20260812 00:12:00.585758 8500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.586287 8488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:00.586566 8488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:00.587723 8488 hybrid_clock.cc:648] HybridClock initialized: now 1786493520587702 us; error 36 us; skew 500 ppm
I20260812 00:12:00.588939 8488 webserver.cc:460] Webserver started at http://127.31.227.62:38993/ using document root <none> and password file <none>
I20260812 00:12:00.589200 8488 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:00.589246 8488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:00.589366 8488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:00.590435 8488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "c9d03ae7c126430280005fef333a8003"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "d3c8f47e1f2e74dee1d91f106b48fdcc"
server_key_iv: "398505cec24e1c391ada546c4b5332d1"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.590734 8488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "c9d03ae7c126430280005fef333a8003"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "d3c8f47e1f2e74dee1d91f106b48fdcc"
server_key_iv: "398505cec24e1c391ada546c4b5332d1"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.591984 8488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:00.592664 8516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.592895 8488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:00.592996 8488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "c9d03ae7c126430280005fef333a8003"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "d3c8f47e1f2e74dee1d91f106b48fdcc"
server_key_iv: "398505cec24e1c391ada546c4b5332d1"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.593075 8488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:00.596510 8488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:00.596804 8488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:00.596959 8488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:00.601199 8488 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:40117
I20260812 00:12:00.601259 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:40117 every 8 connection(s)
I20260812 00:12:00.601640 8488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:00.602324 8602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:00.605090 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003: Bootstrap starting.
I20260812 00:12:00.605752 8602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:00.606072 8602 log.cc:824] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:00.606729 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003: No bootstrap required, opened a new log
I20260812 00:12:00.608345 8602 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d03ae7c126430280005fef333a8003" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40117 } }
I20260812 00:12:00.608489 8602 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:00.608512 8602 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d03ae7c126430280005fef333a8003, State: Initialized, Role: FOLLOWER
I20260812 00:12:00.608685 8602 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d03ae7c126430280005fef333a8003" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40117 } }
I20260812 00:12:00.608745 8602 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:00.608805 8602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:00.608858 8602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:00.609443 8602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d03ae7c126430280005fef333a8003" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40117 } }
I20260812 00:12:00.609472 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 8488
I20260812 00:12:00.609622 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:00.609598 8602 leader_election.cc:302] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d03ae7c126430280005fef333a8003; no voters:
I20260812 00:12:00.609766 8602 leader_election.cc:288] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:00.609817 8605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:00.609859 32652 external_mini_cluster.cc:1428] Setting key f9e2de5435045ef4cbf3353a4162d7e6
I20260812 00:12:00.610010 8605 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [term 1 LEADER]: Becoming Leader. State: Replica: c9d03ae7c126430280005fef333a8003, State: Running, Role: LEADER
I20260812 00:12:00.610085 8602 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:00.610117 8605 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d03ae7c126430280005fef333a8003" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40117 } }
I20260812 00:12:00.610504 8605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d03ae7c126430280005fef333a8003. Latest consensus state: current_term: 1 leader_uuid: "c9d03ae7c126430280005fef333a8003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d03ae7c126430280005fef333a8003" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40117 } } }
I20260812 00:12:00.610590 8605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:00.610864 8606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d03ae7c126430280005fef333a8003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d03ae7c126430280005fef333a8003" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40117 } } }
I20260812 00:12:00.610961 8606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:00.611613 8616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:00.612782 8616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:00.612792 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.610532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.31.227.62:40117)
0812 00:12:00.610760 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.610770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.611178 (+ 408us) server_negotiation.cc:407] Connection header received
0812 00:12:00.611318 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.611327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.611375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.611538 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.612290 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.612294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.612295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:00.612300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.612374 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:00.612389 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.612472 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.612520 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.612579 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260812 00:12:00.614462 8616 catalog_manager.cc:1347] Generated new cluster ID: 324a8d47e6d546f1b871fd2e35d3ff08
I20260812 00:12:00.614530 8616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:00.622037 8616 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:00.622215 8616 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:00.627569 8616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003: Generated new TSK 0
I20260812 00:12:00.627784 8616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:00.631048 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40117
--builtin_ntp_servers=127.31.227.20:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:12:00.717496 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:00.717774 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:00.717813 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:00.718994 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:00.719058 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:00.719103 8640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:00.720479 8640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:00.720741 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:00.721048 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:00.726866 8652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.726882 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.726902 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.727082 8640 server_base.cc:1034] running on GCE node
I20260812 00:12:00.727413 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:00.727628 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:00.728783 8640 hybrid_clock.cc:648] HybridClock initialized: now 1786493520728723 us; error 69 us; skew 500 ppm
I20260812 00:12:00.730118 8640 webserver.cc:460] Webserver started at http://127.31.227.1:38669/ using document root <none> and password file <none>
I20260812 00:12:00.730369 8640 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:00.730453 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:00.730638 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:00.731747 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "1f0fb8b063f141c5ab17d55dc35e46c1"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "8cc8025a7dd012f83e6297acd8845ba4"
server_key_iv: "cc032096a5e70f127769d393c801930a"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.732148 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "1f0fb8b063f141c5ab17d55dc35e46c1"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "8cc8025a7dd012f83e6297acd8845ba4"
server_key_iv: "cc032096a5e70f127769d393c801930a"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.733640 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:12:00.734511 8666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.734697 8640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:00.734870 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "1f0fb8b063f141c5ab17d55dc35e46c1"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "8cc8025a7dd012f83e6297acd8845ba4"
server_key_iv: "cc032096a5e70f127769d393c801930a"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.734978 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:00.738504 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:00.738790 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:00.738946 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:00.739179 8640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:00.739508 8640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:00.739562 8640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.739605 8640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:00.739624 8640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.745755 8640 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:42671
I20260812 00:12:00.745821 8868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:42671 every 8 connection(s)
I20260812 00:12:00.746134 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:00.746925 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 8640
I20260812 00:12:00.747043 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:00.747246 32652 external_mini_cluster.cc:1428] Setting key a6e2287057fa38d21448bd86f2ae718e
I20260812 00:12:00.748297 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40117
--builtin_ntp_servers=127.31.227.20:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 00:12:00.749567 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.746947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:39909 (local address 127.31.227.62:40117)
0812 00:12:00.747009 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.747012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.747312 (+ 300us) server_negotiation.cc:407] Connection header received
0812 00:12:00.747943 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.747944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.747951 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.748028 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.749268 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.749269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.749270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:00.749275 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.749293 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:00.749298 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.749363 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.749417 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.749435 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 00:12:00.749774 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.746939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40117 (local address 127.31.227.1:39909)
0812 00:12:00.747171 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.747213 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.747810 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.748089 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.748093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.748101 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:00.749112 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:00.749113 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:00.749142 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:00.749173 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:00.749364 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:00.749367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:00.749369 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.749423 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.749473 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260812 00:12:00.750329 8869 heartbeater.cc:347] Connected to a master server at 127.31.227.62:40117
I20260812 00:12:00.750442 8869 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:00.750622 8869 heartbeater.cc:511] Master 127.31.227.62:40117 requested a full tablet report, sending...
I20260812 00:12:00.751206 8539 ts_manager.cc:194] Registered new tserver with Master: 1f0fb8b063f141c5ab17d55dc35e46c1 (127.31.227.1:42671)
W20260812 00:12:00.833411 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:00.833690 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:00.833717 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:00.834841 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:00.834875 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:00.834900 8874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:00.836176 8874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:00.836360 8874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:00.836604 8874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:00.842159 8887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.842278 8874 server_base.cc:1034] running on GCE node
W20260812 00:12:00.842139 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.842367 8889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.842599 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:00.842883 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:00.844063 8874 hybrid_clock.cc:648] HybridClock initialized: now 1786493520844048 us; error 27 us; skew 500 ppm
I20260812 00:12:00.845409 8874 webserver.cc:460] Webserver started at http://127.31.227.2:45097/ using document root <none> and password file <none>
I20260812 00:12:00.845722 8874 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:00.845798 8874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:00.845958 8874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:00.847054 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "b67ccd5725b24103a6e4fc9205de38f1"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "6e3d7e745edb26c002c70014efd1824a"
server_key_iv: "43cf83019f92061a2128bbc531e9147f"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.847481 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "b67ccd5725b24103a6e4fc9205de38f1"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "6e3d7e745edb26c002c70014efd1824a"
server_key_iv: "43cf83019f92061a2128bbc531e9147f"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.848870 8874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:00.849624 8902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.849812 8874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:00.849908 8874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "b67ccd5725b24103a6e4fc9205de38f1"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "6e3d7e745edb26c002c70014efd1824a"
server_key_iv: "43cf83019f92061a2128bbc531e9147f"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.849968 8874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:00.853307 8874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:00.853613 8874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:00.853721 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:00.853889 8874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:00.854156 8874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:00.854185 8874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.854205 8874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:00.854218 8874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.860050 8874 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:36587
I20260812 00:12:00.860185 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:36587 every 8 connection(s)
I20260812 00:12:00.860430 8874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:00.863499 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.861188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60933 (local address 127.31.227.62:40117)
0812 00:12:00.861277 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.861280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.861486 (+ 206us) server_negotiation.cc:407] Connection header received
0812 00:12:00.862030 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.862031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.862038 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.862093 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.863307 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.863308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.863308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:00.863311 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.863323 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:00.863327 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.863369 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.863428 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.863440 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:12:00.863750 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.861146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40117 (local address 127.31.227.2:60933)
0812 00:12:00.861353 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.861386 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.861936 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.862147 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.862149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.862157 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:00.863171 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:00.863171 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:00.863200 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:00.863230 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:00.863367 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:00.863370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:00.863372 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.863427 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.863512 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260812 00:12:00.864313 9112 heartbeater.cc:347] Connected to a master server at 127.31.227.62:40117
I20260812 00:12:00.864441 9112 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:00.864609 9112 heartbeater.cc:511] Master 127.31.227.62:40117 requested a full tablet report, sending...
I20260812 00:12:00.864657 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 8874
I20260812 00:12:00.864794 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:00.865031 32652 external_mini_cluster.cc:1428] Setting key 4417545e74f10cea28ed2a3ec5fba860
I20260812 00:12:00.865038 8539 ts_manager.cc:194] Registered new tserver with Master: b67ccd5725b24103a6e4fc9205de38f1 (127.31.227.2:36587)
I20260812 00:12:00.866086 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40117
--builtin_ntp_servers=127.31.227.20:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:12:00.951678 9116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:00.951897 9116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:00.951920 9116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:00.953158 9116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:00.953202 9116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:00.953228 9116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:00.954682 9116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:00.954949 9116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:00.955256 9116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:00.960521 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:00.960534 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.960569 9116 server_base.cc:1034] running on GCE node
W20260812 00:12:00.960583 9126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:00.960928 9116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:00.961212 9116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:00.962393 9116 hybrid_clock.cc:648] HybridClock initialized: now 1786493520962369 us; error 53 us; skew 500 ppm
I20260812 00:12:00.963701 9116 webserver.cc:460] Webserver started at http://127.31.227.3:43923/ using document root <none> and password file <none>
I20260812 00:12:00.963963 9116 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:00.964035 9116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:00.964193 9116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:00.965314 9116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "03ffff58b42c4f4fb8df0298991a097c"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "cc06b4d77d326cab818d8349fc262607"
server_key_iv: "29243a4634a27bc03cf8eb3662293391"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.965714 9116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "03ffff58b42c4f4fb8df0298991a097c"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "cc06b4d77d326cab818d8349fc262607"
server_key_iv: "29243a4634a27bc03cf8eb3662293391"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.967090 9116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:00.967849 9141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.968029 9116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:00.968132 9116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "03ffff58b42c4f4fb8df0298991a097c"
format_stamp: "Formatted at 2026-08-12 00:12:00 on dist-test-slave-zr1t"
server_key: "cc06b4d77d326cab818d8349fc262607"
server_key_iv: "29243a4634a27bc03cf8eb3662293391"
server_key_version: "encryptionkey@0"
I20260812 00:12:00.968250 9116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:00.971750 9116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:00.972034 9116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:00.972182 9116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:00.972404 9116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:00.972730 9116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:00.972786 9116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.972826 9116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:00.972854 9116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:00.979092 9116 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:40219
I20260812 00:12:00.979173 9313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:40219 every 8 connection(s)
I20260812 00:12:00.979493 9116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:00.981920 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 9116
I20260812 00:12:00.982040 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:00.982236 32652 external_mini_cluster.cc:1428] Setting key e62c9efd57184681aba7a963d60c0c2d
I20260812 00:12:00.982970 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.980299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:40717 (local address 127.31.227.62:40117)
0812 00:12:00.980377 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.980379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.980618 (+ 239us) server_negotiation.cc:407] Connection header received
0812 00:12:00.981092 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.981094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.981099 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.981173 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.982692 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.982694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.982694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:00.982699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.982717 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:00.982722 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.982776 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.982879 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.982897 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:12:00.983173 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.980268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40117 (local address 127.31.227.3:40717)
0812 00:12:00.980483 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:00.980524 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:00.981002 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:00.981228 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:00.981231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:00.981239 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:00.982504 (+ 1265us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:00.982505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:00.982544 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:00.982580 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:00.982785 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:00.982793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:00.982796 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:00.982871 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 00:12:00.982944 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260812 00:12:00.983645 9316 heartbeater.cc:347] Connected to a master server at 127.31.227.62:40117
I20260812 00:12:00.983754 9316 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:00.983909 9316 heartbeater.cc:511] Master 127.31.227.62:40117 requested a full tablet report, sending...
I20260812 00:12:00.984275 8539 ts_manager.cc:194] Registered new tserver with Master: 03ffff58b42c4f4fb8df0298991a097c (127.31.227.3:40219)
I20260812 00:12:00.994262 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:00.996969 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:00.995696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.31.227.62:40117)
0812 00:12:00.995755 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:00.995756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:00.995923 (+ 167us) server_negotiation.cc:407] Connection header received
0812 00:12:00.996005 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:00.996006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:00.996012 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:00.996057 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:00.996736 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:00.996738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:00.996738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:00.996743 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:00.996759 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:00.996763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:00.996817 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:00.996880 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 00:12:00.996895 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:12:00.998751 8539 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:01.000309 8539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:01.004206 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.002617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:42671 (local address 127.0.0.1:40446)
0812 00:12:01.002952 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.002962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.002997 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.003230 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.003237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.003244 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:01.003758 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:01.003759 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:01.003775 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:01.003797 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:01.004038 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:01.004045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:01.004049 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:01.004121 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:12:01.004137 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260812 00:12:01.004228 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.002851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.31.227.1:42671)
0812 00:12:01.003073 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.003078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.003089 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:01.003120 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.003122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.003132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.003213 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.003891 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.003899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.003899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.003904 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.003946 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.003968 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.004044 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.004129 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.004145 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260812 00:12:01.004995 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.002617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:36587 (local address 127.0.0.1:60218)
0812 00:12:01.002861 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.002902 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.002968 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.004196 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.004197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.004200 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:01.004590 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:01.004591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:01.004597 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:01.004609 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:01.004868 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:01.004869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:01.004870 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:12:01.004928 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 00:12:01.004941 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":64,"threads_started":1}
I20260812 00:12:01.005035 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.002728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.31.227.2:36587)
0812 00:12:01.003606 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.003611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.003622 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:01.003649 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.003651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.003661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.003739 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.004710 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.004722 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.004722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.004727 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.004768 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.004789 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.004870 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.004934 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.004950 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":122,"threads_started":1}
I20260812 00:12:01.005290 8738 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce079a98ae3490fb5a87609ff5a7321 (DEFAULT_TABLE table=test-table [id=a1c8e385f1714d96a0c649bdd0d3a1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:01.005730 8738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce079a98ae3490fb5a87609ff5a7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:01.005790 8996 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce079a98ae3490fb5a87609ff5a7321 (DEFAULT_TABLE table=test-table [id=a1c8e385f1714d96a0c649bdd0d3a1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:01.006148 8996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce079a98ae3490fb5a87609ff5a7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:01.009313 9343 tablet_bootstrap.cc:492] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Bootstrap starting.
I20260812 00:12:01.009641 9342 tablet_bootstrap.cc:492] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Bootstrap starting.
I20260812 00:12:01.009824 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.002827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:40219 (local address 127.0.0.1:33236)
0812 00:12:01.004027 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.004036 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.004110 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.004575 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.004577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.004579 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:01.009321 (+ 4742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:01.009322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:01.009329 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:01.009342 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:01.009679 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:01.009680 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:01.009680 (+ 0us) client_negotiation.cc:769] Sending connection context
0812 00:12:01.009741 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:12:01.009755 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":74,"threads_started":1}
I20260812 00:12:01.009853 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.003008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.31.227.3:40219)
0812 00:12:01.003213 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.003218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.004111 (+ 893us) server_negotiation.cc:407] Connection header received
0812 00:12:01.004184 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.004188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.004197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.004286 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.009491 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.009502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.009503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.009507 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.009560 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.009592 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.009685 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.009750 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.009766 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260812 00:12:01.010334 9342 tablet_bootstrap.cc:654] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:01.010469 9220 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce079a98ae3490fb5a87609ff5a7321 (DEFAULT_TABLE table=test-table [id=a1c8e385f1714d96a0c649bdd0d3a1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:01.010834 9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce079a98ae3490fb5a87609ff5a7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:01.011159 9343 tablet_bootstrap.cc:654] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:01.011456 9343 log.cc:824] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:01.011926 9342 log.cc:824] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:01.012506 9343 tablet_bootstrap.cc:492] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: No bootstrap required, opened a new log
I20260812 00:12:01.012588 9343 ts_tablet_manager.cc:1397] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:01.013859 9349 tablet_bootstrap.cc:492] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Bootstrap starting.
I20260812 00:12:01.014072 9342 tablet_bootstrap.cc:492] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: No bootstrap required, opened a new log
I20260812 00:12:01.014184 9342 ts_tablet_manager.cc:1397] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260812 00:12:01.014675 9349 tablet_bootstrap.cc:654] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:01.014987 9349 log.cc:824] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:01.015010 9343 raft_consensus.cc:348] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.015185 9343 raft_consensus.cc:374] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:01.015231 9343 raft_consensus.cc:729] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0fb8b063f141c5ab17d55dc35e46c1, State: Initialized, Role: FOLLOWER
I20260812 00:12:01.015367 9343 consensus_queue.cc:260] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.015663 9343 ts_tablet_manager.cc:1428] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:01.015704 8869 heartbeater.cc:503] Master 127.31.227.62:40117 was elected leader, sending a full tablet report...
I20260812 00:12:01.015784 9349 tablet_bootstrap.cc:492] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: No bootstrap required, opened a new log
I20260812 00:12:01.015836 9349 ts_tablet_manager.cc:1397] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:01.016955 9342 raft_consensus.cc:348] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.017109 9342 raft_consensus.cc:374] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:01.017153 9342 raft_consensus.cc:729] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67ccd5725b24103a6e4fc9205de38f1, State: Initialized, Role: FOLLOWER
I20260812 00:12:01.017282 9342 consensus_queue.cc:260] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.017558 9342 ts_tablet_manager.cc:1428] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:12:01.017627 9112 heartbeater.cc:503] Master 127.31.227.62:40117 was elected leader, sending a full tablet report...
I20260812 00:12:01.018502 9349 raft_consensus.cc:348] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.018673 9349 raft_consensus.cc:374] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:01.018708 9349 raft_consensus.cc:729] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ffff58b42c4f4fb8df0298991a097c, State: Initialized, Role: FOLLOWER
I20260812 00:12:01.018822 9349 consensus_queue.cc:260] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.019086 9349 ts_tablet_manager.cc:1428] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:12:01.019143 9316 heartbeater.cc:503] Master 127.31.227.62:40117 was elected leader, sending a full tablet report...
W20260812 00:12:01.111011 9114 tablet.cc:2369] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:01.230124 9322 tablet.cc:2369] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:01.246917 8871 tablet.cc:2369] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:01.270283 9354 raft_consensus.cc:482] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:01.270512 9354 raft_consensus.cc:504] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.271010 9354 leader_election.cc:288] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b67ccd5725b24103a6e4fc9205de38f1 (127.31.227.2:36587), 03ffff58b42c4f4fb8df0298991a097c (127.31.227.3:40219)
I20260812 00:12:01.272565 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.271320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:59149 (local address 127.31.227.3:40219)
0812 00:12:01.271370 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.271372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.271439 (+ 67us) server_negotiation.cc:407] Connection header received
0812 00:12:01.271528 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.271529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.271536 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.271616 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.272311 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.272313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.272313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.272318 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.272337 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.272342 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.272401 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.272461 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.272475 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:01.272614 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.271222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:40219 (local address 127.31.227.1:59149)
0812 00:12:01.271364 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.271376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.271458 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.271649 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.271650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.271654 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:01.272216 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:01.272216 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:01.272223 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:01.272236 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:01.272409 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:01.272410 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:01.272411 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:12:01.272466 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:12:01.272481 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260812 00:12:01.272712 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.271390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:35889 (local address 127.31.227.2:36587)
0812 00:12:01.271450 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.271452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.271460 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:01.271513 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.271514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.271520 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.271588 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.272432 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.272433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.272434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.272439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.272474 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.272479 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.272536 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.272624 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.272639 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:01.272784 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.271222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:36587 (local address 127.31.227.1:35889)
0812 00:12:01.271364 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.271376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.271433 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.271617 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.271619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.271621 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:01.272287 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:01.272287 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:01.272296 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:01.272320 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:01.272546 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:01.272547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:01.272548 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:12:01.272605 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:12:01.272619 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260812 00:12:01.273084 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce079a98ae3490fb5a87609ff5a7321" candidate_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ffff58b42c4f4fb8df0298991a097c" is_pre_election: true
I20260812 00:12:01.273178 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce079a98ae3490fb5a87609ff5a7321" candidate_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67ccd5725b24103a6e4fc9205de38f1" is_pre_election: true
I20260812 00:12:01.273275 9251 raft_consensus.cc:2454] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0fb8b063f141c5ab17d55dc35e46c1 in term 0.
I20260812 00:12:01.273314 9035 raft_consensus.cc:2454] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0fb8b063f141c5ab17d55dc35e46c1 in term 0.
I20260812 00:12:01.273578 8670 leader_election.cc:302] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ffff58b42c4f4fb8df0298991a097c, 1f0fb8b063f141c5ab17d55dc35e46c1; no voters:
I20260812 00:12:01.273761 9354 raft_consensus.cc:2781] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:01.273809 9354 raft_consensus.cc:482] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:01.273825 9354 raft_consensus.cc:3037] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:01.274544 9354 raft_consensus.cc:504] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.274691 9354 leader_election.cc:288] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [CANDIDATE]: Term 1 election: Requested vote from peers b67ccd5725b24103a6e4fc9205de38f1 (127.31.227.2:36587), 03ffff58b42c4f4fb8df0298991a097c (127.31.227.3:40219)
I20260812 00:12:01.274837 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce079a98ae3490fb5a87609ff5a7321" candidate_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67ccd5725b24103a6e4fc9205de38f1"
I20260812 00:12:01.274834 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce079a98ae3490fb5a87609ff5a7321" candidate_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ffff58b42c4f4fb8df0298991a097c"
I20260812 00:12:01.274930 9035 raft_consensus.cc:3037] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:01.274927 9251 raft_consensus.cc:3037] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:01.275637 9035 raft_consensus.cc:2454] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0fb8b063f141c5ab17d55dc35e46c1 in term 1.
I20260812 00:12:01.275699 9251 raft_consensus.cc:2454] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0fb8b063f141c5ab17d55dc35e46c1 in term 1.
I20260812 00:12:01.275870 8669 leader_election.cc:302] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0fb8b063f141c5ab17d55dc35e46c1, b67ccd5725b24103a6e4fc9205de38f1; no voters:
I20260812 00:12:01.276001 9354 raft_consensus.cc:2781] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:01.276182 9354 raft_consensus.cc:686] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0fb8b063f141c5ab17d55dc35e46c1, State: Running, Role: LEADER
I20260812 00:12:01.276294 9354 consensus_queue.cc:237] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } }
I20260812 00:12:01.277158 8539 catalog_manager.cc:5495] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0fb8b063f141c5ab17d55dc35e46c1 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0fb8b063f141c5ab17d55dc35e46c1" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 42671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 } health_report { overall_health: UNKNOWN } } }
I20260812 00:12:01.287554 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.286485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.31.227.1:42671)
0812 00:12:01.286536 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.286538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.286546 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:01.286607 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.286608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.286621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.286668 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.287399 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.287400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.287400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.287403 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.287416 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.287419 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.287464 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.287495 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.287505 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:12:01.289390 9035 raft_consensus.cc:1261] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0fb8b063f141c5ab17d55dc35e46c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:01.289388 9251 raft_consensus.cc:1261] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 1 FOLLOWER]: Refusing update from remote peer 1f0fb8b063f141c5ab17d55dc35e46c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:01.289778 9364 consensus_queue.cc:1035] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67ccd5725b24103a6e4fc9205de38f1" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:01.289927 9354 consensus_queue.cc:1035] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ffff58b42c4f4fb8df0298991a097c" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:01.292583 9372 mvcc.cc:205] Tried to move back new op lower bound from 7317477463198261248 to 7317477463148294144. Current Snapshot: MvccSnapshot[applied={T|T < 7317477463198261248}]
I20260812 00:12:01.295035 8539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:01.295137 8539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:01.295913 8539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9d03ae7c126430280005fef333a8003: Sending DeleteTablet for 3 replicas of tablet 5ce079a98ae3490fb5a87609ff5a7321
I20260812 00:12:01.296322 8738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce079a98ae3490fb5a87609ff5a7321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:01 UTC) from {username='slave'} at 127.0.0.1:40446
I20260812 00:12:01.296356 8996 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce079a98ae3490fb5a87609ff5a7321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:01 UTC) from {username='slave'} at 127.0.0.1:60218
I20260812 00:12:01.296563 9389 tablet_replica.cc:330] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: stopping tablet replica
I20260812 00:12:01.296587 9220 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce079a98ae3490fb5a87609ff5a7321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:01 UTC) from {username='slave'} at 127.0.0.1:33236
I20260812 00:12:01.296648 9389 raft_consensus.cc:2229] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:01.296761 9389 raft_consensus.cc:2258] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:01.296922 9394 tablet_replica.cc:330] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: stopping tablet replica
I20260812 00:12:01.297024 9394 raft_consensus.cc:2229] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:01.297127 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 8640
I20260812 00:12:01.297158 9394 raft_consensus.cc:2258] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:01.297247 9389 ts_tablet_manager.cc:1905] T 5ce079a98ae3490fb5a87609ff5a7321 P 1f0fb8b063f141c5ab17d55dc35e46c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:01.297971 9394 ts_tablet_manager.cc:1905] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:01.299461 9394 ts_tablet_manager.cc:1918] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:01.299546 9394 log.cc:1196] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/5ce079a98ae3490fb5a87609ff5a7321
I20260812 00:12:01.299753 9394 ts_tablet_manager.cc:1939] T 5ce079a98ae3490fb5a87609ff5a7321 P 03ffff58b42c4f4fb8df0298991a097c: Deleting consensus metadata
I20260812 00:12:01.300088 8521 catalog_manager.cc:4841] TS 03ffff58b42c4f4fb8df0298991a097c (127.31.227.3:40219): tablet 5ce079a98ae3490fb5a87609ff5a7321 (table test-table [id=a1c8e385f1714d96a0c649bdd0d3a1d2]) successfully deleted
I20260812 00:12:01.303200 9390 tablet_replica.cc:330] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: stopping tablet replica
I20260812 00:12:01.303334 9390 raft_consensus.cc:2229] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:01.303485 9390 raft_consensus.cc:2258] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:01.306121 9390 ts_tablet_manager.cc:1905] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:01.307565 9390 ts_tablet_manager.cc:1918] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:01.307633 9390 log.cc:1196] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/5ce079a98ae3490fb5a87609ff5a7321
I20260812 00:12:01.307907 9390 ts_tablet_manager.cc:1939] T 5ce079a98ae3490fb5a87609ff5a7321 P b67ccd5725b24103a6e4fc9205de38f1: Deleting consensus metadata
I20260812 00:12:01.308357 8519 catalog_manager.cc:4841] TS b67ccd5725b24103a6e4fc9205de38f1 (127.31.227.2:36587): tablet 5ce079a98ae3490fb5a87609ff5a7321 (table test-table [id=a1c8e385f1714d96a0c649bdd0d3a1d2]) successfully deleted
W20260812 00:12:01.308938 8521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:42671 (error 108)
I20260812 00:12:01.309432 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 8874
I20260812 00:12:01.309468 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.309309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:42671 (local address 127.0.0.1:40456)
0812 00:12:01.309382 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.309441 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:42671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 00:12:01.316690 8521 catalog_manager.cc:4584] TS 1f0fb8b063f141c5ab17d55dc35e46c1 (127.31.227.1:42671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ce079a98ae3490fb5a87609ff5a7321: Network error: Client connection negotiation failed: client connection to 127.31.227.1:42671: connect: Connection refused (error 111)
I20260812 00:12:01.320775 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 9116
I20260812 00:12:01.328349 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 8488
2026-08-12T00:12:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (867 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 00:12:01.354461 32652 mini_oidc.cc:152] Starting JWKS server
I20260812 00:12:01.356174 32652 webserver.cc:460] Webserver started at https://127.0.0.1:44001/ using document root <none> and password file <none>
I20260812 00:12:01.356259 32652 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:12:01.356413 32652 webserver.cc:460] Webserver started at http://127.0.0.1:37223/ using document root <none> and password file <none>
2026-08-12T00:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:01Z Disabled control of system clock
I20260812 00:12:01.365419 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40097
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:40097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/testchainca.cert with env {}
W20260812 00:12:01.449559 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:01.449801 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:01.449836 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:01.450986 9409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:01.451057 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:01.451079 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:01.451099 9409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:01.451131 9409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:12:01.451155 9409 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:12:01.451176 9409 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44001/jwks.json
I20260812 00:12:01.452759 9409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:40097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:01.453038 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:01.453258 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:01.458285 9422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:01.458259 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:01.458285 9414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:01.458411 9409 server_base.cc:1034] running on GCE node
I20260812 00:12:01.458820 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:01.459134 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:01.460275 9409 hybrid_clock.cc:648] HybridClock initialized: now 1786493521460261 us; error 30 us; skew 500 ppm
I20260812 00:12:01.461615 9409 webserver.cc:460] Webserver started at http://127.31.227.62:36209/ using document root <none> and password file <none>
I20260812 00:12:01.461881 9409 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:01.461930 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:01.462082 9409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:01.463177 9409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "c2377ad38a3241eaabbf29c62514f662"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "88cefbe8fefb0a6b5dbc20d33e9e068e"
server_key_iv: "5dd3905f5be8e4a108d2ee8e07875558"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.463552 9409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "c2377ad38a3241eaabbf29c62514f662"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "88cefbe8fefb0a6b5dbc20d33e9e068e"
server_key_iv: "5dd3905f5be8e4a108d2ee8e07875558"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.464861 9409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:12:01.465649 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:01.465827 9409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:01.465922 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "c2377ad38a3241eaabbf29c62514f662"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "88cefbe8fefb0a6b5dbc20d33e9e068e"
server_key_iv: "5dd3905f5be8e4a108d2ee8e07875558"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.466010 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:01.483662 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:01.484007 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:01.484162 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:01.488513 9409 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:40097
I20260812 00:12:01.488552 9513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:40097 every 8 connection(s)
I20260812 00:12:01.488909 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:01.489964 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 9409
I20260812 00:12:01.490000 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:01.490052 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:01.490252 32652 external_mini_cluster.cc:1428] Setting key a2e4d1c2d4d1204177960af914b42ca4
I20260812 00:12:01.493618 9514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662: Bootstrap starting.
I20260812 00:12:01.494287 9514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:01.494596 9514 log.cc:824] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:01.495436 9514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662: No bootstrap required, opened a new log
I20260812 00:12:01.496479 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.490812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.31.227.62:40097)
0812 00:12:01.491054 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.491063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.491101 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:12:01.491643 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.491654 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.491741 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.491905 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.492088 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.493452 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.494306 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.494908 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.495001 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:01.495753 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.495757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.495757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.495965 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.496009 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.496023 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.496095 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.496160 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.496242 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260812 00:12:01.497748 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:12:01.497996 9514 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2377ad38a3241eaabbf29c62514f662" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40097 } }
I20260812 00:12:01.498164 9514 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:01.498201 9514 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2377ad38a3241eaabbf29c62514f662, State: Initialized, Role: FOLLOWER
I20260812 00:12:01.498330 9514 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2377ad38a3241eaabbf29c62514f662" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40097 } }
I20260812 00:12:01.498404 9514 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:01.498437 9514 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:01.498490 9514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:01.499078 9514 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2377ad38a3241eaabbf29c62514f662" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40097 } }
I20260812 00:12:01.499197 9514 leader_election.cc:302] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2377ad38a3241eaabbf29c62514f662; no voters:
I20260812 00:12:01.499366 9514 leader_election.cc:288] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:01.499445 9527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:01.499652 9514 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:01.499658 9527 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [term 1 LEADER]: Becoming Leader. State: Replica: c2377ad38a3241eaabbf29c62514f662, State: Running, Role: LEADER
I20260812 00:12:01.499825 9527 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2377ad38a3241eaabbf29c62514f662" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40097 } }
I20260812 00:12:01.500216 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2377ad38a3241eaabbf29c62514f662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2377ad38a3241eaabbf29c62514f662" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40097 } } }
I20260812 00:12:01.500240 9530 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2377ad38a3241eaabbf29c62514f662. Latest consensus state: current_term: 1 leader_uuid: "c2377ad38a3241eaabbf29c62514f662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2377ad38a3241eaabbf29c62514f662" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40097 } } }
I20260812 00:12:01.500306 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:01.500360 9530 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:01.500543 9541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:01.501156 9541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:01.505947 9541 catalog_manager.cc:1347] Generated new cluster ID: ccd905472ce942f6b256c361b69dae2e
I20260812 00:12:01.506013 9541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:01.530650 9541 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:01.531401 9541 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:01.538784 9541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2377ad38a3241eaabbf29c62514f662: Generated new TSK 0
I20260812 00:12:01.538990 9541 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 00:12:01.574203 9568 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 00:12:01.574345 9568 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:12:01.574628 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.551455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.31.227.62:40097)
0812 00:12:01.551524 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.551527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.551662 (+ 135us) server_negotiation.cc:407] Connection header received
0812 00:12:01.551766 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.551766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.551778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.551834 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:12:01.551981 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.552465 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.553662 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.553862 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.553929 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:01.554173 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:12:01.574164 (+ 19991us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 00:12:01.574562 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49874: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 00:12:01.574715 9523 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49874: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 00:12:01.574913 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:01.551329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40097 (local address 127.0.0.1:49874)
0812 00:12:01.551582 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.551594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.551670 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.551849 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.551851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.551852 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:12:01.551921 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:01.551923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.552568 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.552569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:01.553543 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:01.553546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.553955 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.553956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:01.554023 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:01.554046 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:12:01.574317 (+ 20271us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 00:12:01.574460 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.227.62:40097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":148,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.227.62:40097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 00:12:01.575389 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 9409
2026-08-12T00:12:01Z chronyd exiting
I20260812 00:12:01.621758 32652 test_util.cc:175] -----------------------------------------------
I20260812 00:12:01.621860 32652 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493496733958-32652-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (268 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 00:12:01.622704 32652 mini_oidc.cc:152] Starting JWKS server
I20260812 00:12:01.623450 32652 webserver.cc:460] Webserver started at https://127.0.0.1:34165/ using document root <none> and password file <none>
I20260812 00:12:01.623530 32652 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:12:01.623684 32652 webserver.cc:460] Webserver started at http://127.0.0.1:42015/ using document root <none> and password file <none>
2026-08-12T00:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:01Z Disabled control of system clock
I20260812 00:12:01.632797 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45233
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:45233
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34165/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/testchainca.cert with env {}
W20260812 00:12:01.714715 9575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:01.714929 9575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:01.714952 9575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:01.716128 9575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:01.716171 9575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:01.716184 9575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:01.716197 9575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:01.716251 9575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:12:01.716269 9575 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:12:01.716293 9575 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34165/jwks.json
I20260812 00:12:01.717855 9575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:45233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34165/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:01.718134 9575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:01.718405 9575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:01.723376 9595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:01.723366 9586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:01.723379 9588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:01.723397 9575 server_base.cc:1034] running on GCE node
I20260812 00:12:01.723904 9575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:01.724228 9575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:01.725414 9575 hybrid_clock.cc:648] HybridClock initialized: now 1786493521725387 us; error 37 us; skew 500 ppm
I20260812 00:12:01.726842 9575 webserver.cc:460] Webserver started at http://127.31.227.62:34073/ using document root <none> and password file <none>
I20260812 00:12:01.727128 9575 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:01.727205 9575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:01.727371 9575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:01.728452 9575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "1792db7df48b4b1580afc9d30ab68441"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "aeb7314d02a0763137456bfd7851b2fc"
server_key_iv: "3202c54b3d6292738c5610e843d29cf8"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.728857 9575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "1792db7df48b4b1580afc9d30ab68441"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "aeb7314d02a0763137456bfd7851b2fc"
server_key_iv: "3202c54b3d6292738c5610e843d29cf8"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.730360 9575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:01.731199 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:01.731447 9575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:01.731564 9575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "1792db7df48b4b1580afc9d30ab68441"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "aeb7314d02a0763137456bfd7851b2fc"
server_key_iv: "3202c54b3d6292738c5610e843d29cf8"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.731644 9575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:01.751317 9575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:01.751694 9575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:01.751856 9575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:01.756330 9575 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:45233
I20260812 00:12:01.756489 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:45233 every 8 connection(s)
I20260812 00:12:01.756814 9575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:01.757356 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 9575
I20260812 00:12:01.757444 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:01.757629 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:01.757679 32652 external_mini_cluster.cc:1428] Setting key 849d1b67288a5c1b1d6f41d7527b98d6
I20260812 00:12:01.762064 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441: Bootstrap starting.
I20260812 00:12:01.762827 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:01.763182 9684 log.cc:824] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:01.764082 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441: No bootstrap required, opened a new log
I20260812 00:12:01.764410 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.758298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.31.227.62:45233)
0812 00:12:01.758552 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.758562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.758597 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:12:01.759173 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.759184 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.759286 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.759471 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:01.759680 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.761151 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.762065 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.762696 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.762832 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:01.763641 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:01.763645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:01.763646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:01.763868 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:01.763919 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:01.763934 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:01.764004 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:01.764079 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 00:12:01.764159 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260812 00:12:01.765480 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:12:01.766531 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792db7df48b4b1580afc9d30ab68441" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45233 } }
I20260812 00:12:01.766732 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:01.766781 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1792db7df48b4b1580afc9d30ab68441, State: Initialized, Role: FOLLOWER
I20260812 00:12:01.766914 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792db7df48b4b1580afc9d30ab68441" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45233 } }
I20260812 00:12:01.767000 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:01.767043 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:01.767092 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:01.767642 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792db7df48b4b1580afc9d30ab68441" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45233 } }
I20260812 00:12:01.767771 9684 leader_election.cc:302] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1792db7df48b4b1580afc9d30ab68441; no voters:
I20260812 00:12:01.767969 9684 leader_election.cc:288] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:01.768014 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:01.768213 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [term 1 LEADER]: Becoming Leader. State: Replica: 1792db7df48b4b1580afc9d30ab68441, State: Running, Role: LEADER
I20260812 00:12:01.768253 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:01.768301 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792db7df48b4b1580afc9d30ab68441" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45233 } }
I20260812 00:12:01.768668 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1792db7df48b4b1580afc9d30ab68441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792db7df48b4b1580afc9d30ab68441" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45233 } } }
I20260812 00:12:01.768747 9696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1792db7df48b4b1580afc9d30ab68441. Latest consensus state: current_term: 1 leader_uuid: "1792db7df48b4b1580afc9d30ab68441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792db7df48b4b1580afc9d30ab68441" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45233 } } }
I20260812 00:12:01.768754 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:01.768821 9696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:01.769062 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:01.769841 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:01.771306 9700 catalog_manager.cc:1347] Generated new cluster ID: a921d44eefef4d409b6ab46e1474fe12
I20260812 00:12:01.771375 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:01.796593 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:01.797142 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:01.801317 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1792db7df48b4b1580afc9d30ab68441: Generated new TSK 0
I20260812 00:12:01.801550 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 00:12:01.832264 9736 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 00:12:01.832367 9736 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:12:01.832666 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:01.810937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.31.227.62:45233)
0812 00:12:01.811003 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:01.811006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:01.811122 (+ 116us) server_negotiation.cc:407] Connection header received
0812 00:12:01.811226 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:01.811232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:01.811244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:01.811322 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:12:01.811470 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.811951 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.812825 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.813037 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.813117 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:01.813320 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:12:01.832427 (+ 19107us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 00:12:01.832607 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33986: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":39}
W20260812 00:12:01.832743 9690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33986: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 00:12:01.832779 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:01.810872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45233 (local address 127.0.0.1:33986)
0812 00:12:01.811051 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:01.811061 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:01.811142 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:01.811323 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:01.811325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:01.811327 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:12:01.811390 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:01.811392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.812107 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.812109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:01.812730 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:01.812732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:01.813120 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:01.813121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:01.813184 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:01.813190 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:12:01.832551 (+ 19361us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 00:12:01.832682 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.227.62:45233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260812 00:12:01.833099 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 9575
2026-08-12T00:12:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (263 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 00:12:01.886085 32652 mini_oidc.cc:152] Starting JWKS server
I20260812 00:12:01.886869 32652 webserver.cc:460] Webserver started at https://127.0.0.1:41169/ using document root <none> and password file <none>
I20260812 00:12:01.886942 32652 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:12:01.887130 32652 webserver.cc:460] Webserver started at http://127.0.0.1:34445/ using document root <none> and password file <none>
2026-08-12T00:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:01Z Disabled control of system clock
I20260812 00:12:01.896502 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:37601
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:37601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41169/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 00:12:01.979775 9747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:01.979982 9747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:01.980005 9747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:01.981202 9747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:01.981248 9747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:01.981261 9747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:01.981273 9747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:01.981287 9747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:12:01.981299 9747 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:12:01.981310 9747 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41169/jwks.json
I20260812 00:12:01.982671 9747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:37601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41169/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:01.982867 9747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:01.983105 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:01.987990 9758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:01.988014 9747 server_base.cc:1034] running on GCE node
W20260812 00:12:01.987978 9756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:01.987983 9754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:01.988487 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:01.988806 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:01.989971 9747 hybrid_clock.cc:648] HybridClock initialized: now 1786493521989957 us; error 45 us; skew 500 ppm
I20260812 00:12:01.991446 9747 webserver.cc:460] Webserver started at http://127.31.227.62:35865/ using document root <none> and password file <none>
I20260812 00:12:01.991731 9747 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:01.991807 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:01.991971 9747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:01.993058 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "7efb4acb6ede4859b9e74c828f6bded8"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "c6339530bcb96f46f9c4ffb2649ebd64"
server_key_iv: "7b3477a1a5539eb291ea1f5b8a39a256"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.993446 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "7efb4acb6ede4859b9e74c828f6bded8"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "c6339530bcb96f46f9c4ffb2649ebd64"
server_key_iv: "7b3477a1a5539eb291ea1f5b8a39a256"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.995604 9747 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260812 00:12:01.996592 9769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:01.997130 9747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:01.997234 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "7efb4acb6ede4859b9e74c828f6bded8"
format_stamp: "Formatted at 2026-08-12 00:12:01 on dist-test-slave-zr1t"
server_key: "c6339530bcb96f46f9c4ffb2649ebd64"
server_key_iv: "7b3477a1a5539eb291ea1f5b8a39a256"
server_key_version: "encryptionkey@0"
I20260812 00:12:01.997330 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:02.006389 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:02.006707 9747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:02.006863 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:02.010757 9747 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:37601
I20260812 00:12:02.010804 9861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:37601 every 8 connection(s)
I20260812 00:12:02.011179 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:02.011885 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:02.014724 9862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8: Bootstrap starting.
I20260812 00:12:02.015326 9862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:02.015647 9862 log.cc:824] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:02.016351 9862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8: No bootstrap required, opened a new log
I20260812 00:12:02.018070 9862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efb4acb6ede4859b9e74c828f6bded8" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 37601 } }
I20260812 00:12:02.018213 9862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:02.018234 9862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efb4acb6ede4859b9e74c828f6bded8, State: Initialized, Role: FOLLOWER
I20260812 00:12:02.018424 9862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efb4acb6ede4859b9e74c828f6bded8" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 37601 } }
I20260812 00:12:02.018492 9862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:02.018534 9862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:02.018589 9862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:02.019186 9862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efb4acb6ede4859b9e74c828f6bded8" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 37601 } }
I20260812 00:12:02.019300 9862 leader_election.cc:302] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7efb4acb6ede4859b9e74c828f6bded8; no voters:
I20260812 00:12:02.019515 9862 leader_election.cc:288] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:02.019580 9866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:02.019773 9862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:02.019762 9866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [term 1 LEADER]: Becoming Leader. State: Replica: 7efb4acb6ede4859b9e74c828f6bded8, State: Running, Role: LEADER
I20260812 00:12:02.019938 9866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efb4acb6ede4859b9e74c828f6bded8" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 37601 } }
I20260812 00:12:02.020323 9867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7efb4acb6ede4859b9e74c828f6bded8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efb4acb6ede4859b9e74c828f6bded8" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 37601 } } }
I20260812 00:12:02.020464 9867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:02.020507 9870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7efb4acb6ede4859b9e74c828f6bded8. Latest consensus state: current_term: 1 leader_uuid: "7efb4acb6ede4859b9e74c828f6bded8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efb4acb6ede4859b9e74c828f6bded8" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 37601 } } }
I20260812 00:12:02.020573 9870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:02.020996 9879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:02.021339 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 9747
I20260812 00:12:02.021450 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:02.021592 9879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:02.021664 32652 external_mini_cluster.cc:1428] Setting key ec19bf1a9693456cd3eed5984eb4974e
I20260812 00:12:02.023249 9879 catalog_manager.cc:1347] Generated new cluster ID: eeff553636484e92999d06ffed9570dd
I20260812 00:12:02.023305 9879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:02.027238 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:02.022137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.31.227.62:37601)
0812 00:12:02.022343 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:02.022351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:02.022782 (+ 431us) server_negotiation.cc:407] Connection header received
0812 00:12:02.022931 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:02.022937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:02.022992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:02.023132 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:02.023412 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.024300 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.025129 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.025677 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.025792 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:02.026529 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:02.026533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:02.026534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:02.026715 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:02.026748 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:02.026758 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:02.026832 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:02.026925 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 00:12:02.027009 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260812 00:12:02.028173 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:12:02.036866 9879 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:02.037380 9879 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:02.054019 9879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7efb4acb6ede4859b9e74c828f6bded8: Generated new TSK 0
I20260812 00:12:02.054265 9879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:02.070324 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:02.066930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.31.227.62:37601)
0812 00:12:02.066989 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:02.066991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:02.067052 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:12:02.067148 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:02.067150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:02.067162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:02.067249 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:12:02.067390 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.067908 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.068716 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.068960 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.069039 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:02.069228 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:12:02.070084 (+ 856us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 00:12:02.070248 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260812 00:12:02.070392 9900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 00:12:02.070394 9911 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:02.066848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:37601 (local address 127.0.0.1:51688)
0812 00:12:02.066993 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:02.067004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:02.067073 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:02.067247 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:02.067249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:02.067250 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:12:02.067311 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:02.067313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.068005 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.068006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:02.068624 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:02.068626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.069045 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.069046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:02.069106 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:02.069111 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:12:02.070201 (+ 1090us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 00:12:02.070296 (+ 95us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.227.62:37601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260812 00:12:02.073675 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 9747
E20260812 00:12:02.090411 9912 webserver.cc:533] Webserver: sslize failed:
E20260812 00:12:02.090492 9912 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T00:12:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T00:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:02Z Disabled control of system clock
I20260812 00:12:02.130780 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:41897
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:41897
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/testchainca.cert with env {}
W20260812 00:12:02.214306 9920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:02.214530 9920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:02.214552 9920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:02.215828 9920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:02.215874 9920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:02.215888 9920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:02.215900 9920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:02.215941 9920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:12:02.215972 9920 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:12:02.215987 9920 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 00:12:02.217556 9920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:41897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:41897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:02.217808 9920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:02.218034 9920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:02.223039 9935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:02.223064 9931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:02.223096 9932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:02.223256 9920 server_base.cc:1034] running on GCE node
I20260812 00:12:02.223518 9920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:02.223867 9920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:02.225034 9920 hybrid_clock.cc:648] HybridClock initialized: now 1786493522225015 us; error 38 us; skew 500 ppm
I20260812 00:12:02.226310 9920 webserver.cc:460] Webserver started at http://127.31.227.62:44281/ using document root <none> and password file <none>
I20260812 00:12:02.226570 9920 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:02.226641 9920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:02.226794 9920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:02.227883 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "0841e18dbeca4769a6560bb9e9a2f90b"
format_stamp: "Formatted at 2026-08-12 00:12:02 on dist-test-slave-zr1t"
server_key: "a63f732ed47b2751ad5aefb2b1cdaaf6"
server_key_iv: "96e2ccee2e48661c8d12837ade0a836d"
server_key_version: "encryptionkey@0"
I20260812 00:12:02.228327 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "0841e18dbeca4769a6560bb9e9a2f90b"
format_stamp: "Formatted at 2026-08-12 00:12:02 on dist-test-slave-zr1t"
server_key: "a63f732ed47b2751ad5aefb2b1cdaaf6"
server_key_iv: "96e2ccee2e48661c8d12837ade0a836d"
server_key_version: "encryptionkey@0"
I20260812 00:12:02.229712 9920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:12:02.230531 9943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:02.230743 9920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:02.230859 9920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "0841e18dbeca4769a6560bb9e9a2f90b"
format_stamp: "Formatted at 2026-08-12 00:12:02 on dist-test-slave-zr1t"
server_key: "a63f732ed47b2751ad5aefb2b1cdaaf6"
server_key_iv: "96e2ccee2e48661c8d12837ade0a836d"
server_key_version: "encryptionkey@0"
I20260812 00:12:02.230939 9920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:02.255447 9920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:02.255841 9920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:02.256026 9920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:02.260154 9920 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:41897
I20260812 00:12:02.260192 10029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:41897 every 8 connection(s)
I20260812 00:12:02.260579 9920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:02.261683 10030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:02.264072 10030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b: Bootstrap starting.
I20260812 00:12:02.264617 10030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:02.264886 10030 log.cc:824] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:02.265686 10030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b: No bootstrap required, opened a new log
I20260812 00:12:02.265700 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 9920
I20260812 00:12:02.265797 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:02.266004 32652 external_mini_cluster.cc:1428] Setting key 8c155904fe510d7b8770c5989be780dc
I20260812 00:12:02.267693 10030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 41897 } }
I20260812 00:12:02.267841 10030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:02.267865 10030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0841e18dbeca4769a6560bb9e9a2f90b, State: Initialized, Role: FOLLOWER
I20260812 00:12:02.268025 10030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 41897 } }
I20260812 00:12:02.268116 10030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:02.268162 10030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:02.268224 10030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:02.268976 10030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 41897 } }
I20260812 00:12:02.269084 10030 leader_election.cc:302] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0841e18dbeca4769a6560bb9e9a2f90b; no voters:
I20260812 00:12:02.269249 10030 leader_election.cc:288] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:02.269300 10040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:02.269483 10040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [term 1 LEADER]: Becoming Leader. State: Replica: 0841e18dbeca4769a6560bb9e9a2f90b, State: Running, Role: LEADER
I20260812 00:12:02.269502 10030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:02.269591 10040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 41897 } }
I20260812 00:12:02.269944 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 41897 } } }
I20260812 00:12:02.270038 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:02.269933 10040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0841e18dbeca4769a6560bb9e9a2f90b. Latest consensus state: current_term: 1 leader_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e18dbeca4769a6560bb9e9a2f90b" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 41897 } } }
I20260812 00:12:02.270227 10040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b [sys.catalog]: This master's current role is: LEADER
W20260812 00:12:02.271505 10063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:02.271567 10063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:02.271615 10050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:02.271785 10050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:02.273427 10050 catalog_manager.cc:1347] Generated new cluster ID: f2ae527cf7f64fd6b540fae1b148a473
I20260812 00:12:02.273552 10050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:02.273720 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:02.266555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.31.227.62:41897)
0812 00:12:02.266810 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:02.266818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:02.266843 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:12:02.267703 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:02.267711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:02.267782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:02.267960 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:02.268256 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.270095 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.271331 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.271909 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.272060 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:02.272879 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:02.272882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:02.272882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:02.273112 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:02.273153 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:02.273168 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:02.273238 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:02.273355 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 00:12:02.273436 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260812 00:12:02.274647 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:12:02.281114 10050 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:02.281700 10050 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:02.284611 10050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0841e18dbeca4769a6560bb9e9a2f90b: Generated new TSK 0
I20260812 00:12:02.284776 10050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:03.172361 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:02.289439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.31.227.62:41897)
0812 00:12:02.289532 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:02.289534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:02.289672 (+ 138us) server_negotiation.cc:407] Connection header received
0812 00:12:02.289755 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:02.289756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:02.289769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:02.289860 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:12:02.290002 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.290469 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.291268 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.291470 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.291544 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:02.291742 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:12:03.172129 (+880387us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 00:12:03.172298 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43438: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 00:12:03.172469 10039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43438: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 00:12:03.172550 10077 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:02.289352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:41897 (local address 127.0.0.1:43438)
0812 00:12:02.289594 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:02.289604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:02.289692 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:02.289860 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:02.289861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:02.289863 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:12:02.289920 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:02.289922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.290581 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.290582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:02.291179 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:02.291181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:02.291565 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:02.291566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:02.291624 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:02.291631 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:12:03.172281 (+880650us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 00:12:03.172438 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.227.62:41897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":134,"threads_started":1}
I20260812 00:12:03.172917 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 9920
2026-08-12T00:12:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1076 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (93 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:03 dist-test-slave-zr1t krb5kdc[10093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:03 dist-test-slave-zr1t krb5kdc[10093](info): set up 2 sockets
Aug 12 00:12:03 dist-test-slave-zr1t krb5kdc[10093](info): commencing operation
krb5kdc: starting...
W20260812 00:12:04.639842 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:04 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:12:04.768482 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:33371
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:33371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:04.852385 10117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:04.852644 10117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:04.852681 10117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:04.853925 10117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:04.853989 10117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:04.854013 10117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:04.854044 10117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:04.854074 10117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:04.855641 10117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:33371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:04.855937 10117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:04.856177 10117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:04.861335 10123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:04.861378 10124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:04.861335 10126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:04.861586 10117 server_base.cc:1034] running on GCE node
I20260812 00:12:04.861902 10117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:04.862259 10117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:04.863448 10117 hybrid_clock.cc:648] HybridClock initialized: now 1786493524863434 us; error 44 us; skew 500 ppm
Aug 12 00:12:04 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493524, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:04.866202 10117 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:12:04.866719 10117 webserver.cc:460] Webserver started at http://127.31.227.62:38925/ using document root <none> and password file <none>
I20260812 00:12:04.866989 10117 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:04.867058 10117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:04.867220 10117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:04.868330 10117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "8a5fb686231f46bc8bfa8f2e182bd282"
format_stamp: "Formatted at 2026-08-12 00:12:04 on dist-test-slave-zr1t"
server_key: "536d18debec6e3048d83cdb5c89956dc"
server_key_iv: "8145f669f7d74be4ab0fe59e2ca3a670"
server_key_version: "encryptionkey@0"
I20260812 00:12:04.868722 10117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "8a5fb686231f46bc8bfa8f2e182bd282"
format_stamp: "Formatted at 2026-08-12 00:12:04 on dist-test-slave-zr1t"
server_key: "536d18debec6e3048d83cdb5c89956dc"
server_key_iv: "8145f669f7d74be4ab0fe59e2ca3a670"
server_key_version: "encryptionkey@0"
I20260812 00:12:04.870184 10117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:04.871042 10141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:04.871245 10117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:04.871367 10117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "8a5fb686231f46bc8bfa8f2e182bd282"
format_stamp: "Formatted at 2026-08-12 00:12:04 on dist-test-slave-zr1t"
server_key: "536d18debec6e3048d83cdb5c89956dc"
server_key_iv: "8145f669f7d74be4ab0fe59e2ca3a670"
server_key_version: "encryptionkey@0"
I20260812 00:12:04.871449 10117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:04.880467 10117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:04.882957 10117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:04.883133 10117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:04.887574 10117 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:33371
I20260812 00:12:04.887629 10239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:33371 every 8 connection(s)
I20260812 00:12:04.888021 10117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:04.888919 10241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:04.891220 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282: Bootstrap starting.
I20260812 00:12:04.891757 10241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:04.892011 10241 log.cc:824] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:04.892659 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282: No bootstrap required, opened a new log
I20260812 00:12:04.894347 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 10117
I20260812 00:12:04.894474 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:04.894374 10241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33371 } }
I20260812 00:12:04.894568 10241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:04.894614 10241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5fb686231f46bc8bfa8f2e182bd282, State: Initialized, Role: FOLLOWER
I20260812 00:12:04.894714 32652 external_mini_cluster.cc:1428] Setting key 794732f494ecc92ea7a9e79fe2b37cf6
I20260812 00:12:04.894784 10241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33371 } }
I20260812 00:12:04.894872 10241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:04.894925 10241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:04.894986 10241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:04.896148 10241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33371 } }
I20260812 00:12:04.896276 10241 leader_election.cc:302] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a5fb686231f46bc8bfa8f2e182bd282; no voters:
I20260812 00:12:04.896449 10241 leader_election.cc:288] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:04.896514 10248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:04.896682 10248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5fb686231f46bc8bfa8f2e182bd282, State: Running, Role: LEADER
I20260812 00:12:04.896757 10241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:04.896785 10248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33371 } }
I20260812 00:12:04.897238 10248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a5fb686231f46bc8bfa8f2e182bd282. Latest consensus state: current_term: 1 leader_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33371 } } }
I20260812 00:12:04.897321 10248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:04.897459 10250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5fb686231f46bc8bfa8f2e182bd282" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33371 } } }
I20260812 00:12:04.897542 10250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:04.897655 10260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:04.898129 10260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:04.899751 10260 catalog_manager.cc:1347] Generated new cluster ID: b4608c59b31243a49ca4f70d23f0a0a0
I20260812 00:12:04.899806 10260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:12:04 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:04.906843 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:04.895282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.31.227.62:33371)
0812 00:12:04.895506 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:04.895519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:04.895561 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:12:04.895716 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:04.895726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:04.895793 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:04.895953 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:04.896745 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:04.898229 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:04.899168 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:04.899785 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:04.899895 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:04.902932 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:04.902937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:04.902938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:04.902951 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:04.905034 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:04.905440 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:04.905444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:04.905444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:04.905595 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:04.905805 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:04.905806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:04.905807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:04.906199 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:04.906403 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:04.906477 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 00:12:04.906628 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260812 00:12:04.914819 10260 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:04.915408 10260 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:04.920118 10260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a5fb686231f46bc8bfa8f2e182bd282: Generated new TSK 0
I20260812 00:12:04.920320 10260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:12:04.952267 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33371
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:05.039917 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:05.040184 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:05.040225 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:05.041388 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:05.041456 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:05.041563 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:05.043541 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:05.043840 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:05.044123 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:05.051190 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:05.051218 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:05.051265 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:05.051597 10293 server_base.cc:1034] running on GCE node
I20260812 00:12:05.051806 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:05.052062 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:05.053232 10293 hybrid_clock.cc:648] HybridClock initialized: now 1786493525053220 us; error 26 us; skew 500 ppm
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.055976 10293 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:12:05.056450 10293 webserver.cc:460] Webserver started at http://127.31.227.1:40651/ using document root <none> and password file <none>
I20260812 00:12:05.056715 10293 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:05.056787 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:05.056942 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:05.058140 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "9bedc6f599ae4b0a92be30191a92e3bd"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "8beee56918874e0ba8f7aadf11d6b7ff"
server_key_iv: "c073601bf8476871f246eb5ea3a3b8b0"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.058522 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "9bedc6f599ae4b0a92be30191a92e3bd"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "8beee56918874e0ba8f7aadf11d6b7ff"
server_key_iv: "c073601bf8476871f246eb5ea3a3b8b0"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.059897 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:05.060652 10319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.060817 10293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:05.060927 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "9bedc6f599ae4b0a92be30191a92e3bd"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "8beee56918874e0ba8f7aadf11d6b7ff"
server_key_iv: "c073601bf8476871f246eb5ea3a3b8b0"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.061008 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:05.109470 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:05.111541 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:05.111665 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:05.111867 10293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:05.112213 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:05.112249 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.112273 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:05.112331 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.118538 10293 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:33257
I20260812 00:12:05.118597 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:33257 every 8 connection(s)
I20260812 00:12:05.118919 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:05.119163 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 10293
I20260812 00:12:05.119293 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:05.119553 32652 external_mini_cluster.cc:1428] Setting key a1c4cf4332ad642182dd80f53bfc9dd5
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
I20260812 00:12:05.130017 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.119715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:44231 (local address 127.31.227.62:33371)
0812 00:12:05.119793 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.119795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.119992 (+ 197us) server_negotiation.cc:407] Connection header received
0812 00:12:05.121007 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.121008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.121024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.121081 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.121825 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.122639 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.123663 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.124016 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.124093 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.126794 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:05.126795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:05.126796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:05.126803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:05.128371 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.128927 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.128928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.128929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.128980 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.129298 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.129299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.129299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.129432 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:05.129487 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:05.129852 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 00:12:05.129937 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:12:05.130215 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.119703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33371 (local address 127.31.227.1:44231)
0812 00:12:05.119873 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:05.119917 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:05.120882 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:05.121161 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:05.121166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:05.121440 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:05.121711 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:05.121716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.122751 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.122752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:05.123576 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:05.123578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.124110 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.124111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:05.124217 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.124796 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:05.124806 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:05.126682 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:05.128465 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:05.128475 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:05.128478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:05.128845 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:05.129056 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:05.129058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:05.129058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:05.129220 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:05.129508 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:05.129540 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:05.129712 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 00:12:05.129820 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:12:05.129956 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260812 00:12:05.130801 10516 heartbeater.cc:347] Connected to a master server at 127.31.227.62:33371
I20260812 00:12:05.130965 10516 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:05.131242 10516 heartbeater.cc:511] Master 127.31.227.62:33371 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
I20260812 00:12:05.131997 10172 ts_manager.cc:194] Registered new tserver with Master: 9bedc6f599ae4b0a92be30191a92e3bd (127.31.227.1:33257)
I20260812 00:12:05.132711 10172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:44231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:12:05.147975 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33371
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:05.234684 10522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:05.234896 10522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:05.234920 10522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:05.236092 10522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:05.236133 10522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:05.236159 10522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:05.237655 10522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:05.237857 10522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:05.238049 10522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:05.243337 10536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:05.243324 10533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:05.243324 10534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:05.243561 10522 server_base.cc:1034] running on GCE node
I20260812 00:12:05.243835 10522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:05.244110 10522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:05.245286 10522 hybrid_clock.cc:648] HybridClock initialized: now 1786493525245279 us; error 38 us; skew 500 ppm
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.248014 10522 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:12:05.248515 10522 webserver.cc:460] Webserver started at http://127.31.227.2:40871/ using document root <none> and password file <none>
I20260812 00:12:05.248780 10522 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:05.248849 10522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:05.249008 10522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:05.250159 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "81fd2a8c9bb8443ca355e4e57eeb9e83"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "1f5ce4369295b93dda8f4fa30129ce02"
server_key_iv: "b669a60ef398be9f8a92a0cd0eb2a5f2"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.250536 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "81fd2a8c9bb8443ca355e4e57eeb9e83"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "1f5ce4369295b93dda8f4fa30129ce02"
server_key_iv: "b669a60ef398be9f8a92a0cd0eb2a5f2"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.251976 10522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:05.252731 10550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.252914 10522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:05.253026 10522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "81fd2a8c9bb8443ca355e4e57eeb9e83"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "1f5ce4369295b93dda8f4fa30129ce02"
server_key_iv: "b669a60ef398be9f8a92a0cd0eb2a5f2"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.253104 10522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:05.268253 10522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:05.270359 10522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:05.270541 10522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:05.270812 10522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:05.271157 10522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:05.271212 10522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.271255 10522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:05.271283 10522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.277971 10522 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:43117
I20260812 00:12:05.278033 10754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:43117 every 8 connection(s)
I20260812 00:12:05.278354 10522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:05.284366 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 10522
I20260812 00:12:05.284473 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:05.284700 32652 external_mini_cluster.cc:1428] Setting key 3576ce1cb8bf9317f0a565892b03e428
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:05.288769 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.279147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:51833 (local address 127.31.227.62:33371)
0812 00:12:05.279219 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.279222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.279454 (+ 232us) server_negotiation.cc:407] Connection header received
0812 00:12:05.280390 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.280391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.280401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.280464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.281128 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.281754 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.282819 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.283020 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.283075 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.285604 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:05.285606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:05.285606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:05.285615 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:05.287331 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.287755 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.287757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.287757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.287822 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.288084 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.288085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.288086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.288208 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:05.288286 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:05.288580 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:12:05.288677 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:12:05.288954 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.279139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33371 (local address 127.31.227.2:51833)
0812 00:12:05.279312 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:05.279356 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:05.280293 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:05.280519 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:05.280523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:05.280776 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:05.280971 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:05.280975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.281893 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.281894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:05.282735 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:05.282738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.283101 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.283101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:05.283207 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.283683 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:05.283693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:05.285476 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:05.287422 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:05.287429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:05.287431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:05.287652 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:05.287899 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:05.287900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:05.287900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:05.288012 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:05.288288 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:05.288296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:05.288438 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 00:12:05.288543 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:12:05.288707 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260812 00:12:05.289469 10756 heartbeater.cc:347] Connected to a master server at 127.31.227.62:33371
I20260812 00:12:05.289615 10756 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:05.289813 10756 heartbeater.cc:511] Master 127.31.227.62:33371 requested a full tablet report, sending...
I20260812 00:12:05.290258 10172 ts_manager.cc:194] Registered new tserver with Master: 81fd2a8c9bb8443ca355e4e57eeb9e83 (127.31.227.2:43117)
I20260812 00:12:05.290665 10172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:51833
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:12:05.312481 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33371
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:05.398434 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:05.398739 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:05.398769 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:05.399906 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:05.399950 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:05.399976 10764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:05.401405 10764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:33371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:05.401664 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:05.401871 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:05.406904 10773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:05.406960 10774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:05.407034 10764 server_base.cc:1034] running on GCE node
W20260812 00:12:05.407229 10776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:05.407464 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:05.407718 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:05.408888 10764 hybrid_clock.cc:648] HybridClock initialized: now 1786493525408864 us; error 40 us; skew 500 ppm
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.411577 10764 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:12:05.412046 10764 webserver.cc:460] Webserver started at http://127.31.227.3:44669/ using document root <none> and password file <none>
I20260812 00:12:05.412299 10764 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:05.412371 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:05.412528 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:05.413677 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "62d97bf766d74a72a27441c077bb9efe"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "2fbbb5659aeca9a6844cff0ead535cd8"
server_key_iv: "bb335ef6a5c12eecb4b33bdb325490cc"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.414060 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "62d97bf766d74a72a27441c077bb9efe"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "2fbbb5659aeca9a6844cff0ead535cd8"
server_key_iv: "bb335ef6a5c12eecb4b33bdb325490cc"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.415483 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:12:05.416342 10790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.416661 10764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:05.416783 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "62d97bf766d74a72a27441c077bb9efe"
format_stamp: "Formatted at 2026-08-12 00:12:05 on dist-test-slave-zr1t"
server_key: "2fbbb5659aeca9a6844cff0ead535cd8"
server_key_iv: "bb335ef6a5c12eecb4b33bdb325490cc"
server_key_version: "encryptionkey@0"
I20260812 00:12:05.416867 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:05.439254 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:05.441363 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:05.441606 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:05.441874 10764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:05.442219 10764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:05.442277 10764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.442333 10764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:05.442363 10764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:05.448653 10764 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:45663
I20260812 00:12:05.448702 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:45663 every 8 connection(s)
I20260812 00:12:05.449074 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:05.458704 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 10764
I20260812 00:12:05.458820 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:05.458866 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.449857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:46241 (local address 127.31.227.62:33371)
0812 00:12:05.449949 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.449952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.450132 (+ 180us) server_negotiation.cc:407] Connection header received
0812 00:12:05.451088 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.451089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.451100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.451175 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.451865 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.452364 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.453381 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.453635 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.453691 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.455882 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:05.455884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:05.455884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:05.455892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:05.457377 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.457841 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.457842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.457843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.457912 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.458180 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.458181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.458181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.458308 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:05.458382 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:05.458683 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 00:12:05.458772 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 00:12:05.459057 32652 external_mini_cluster.cc:1428] Setting key 05919f4fb0c6838cae66d524877976f2
I20260812 00:12:05.459087 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.449807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33371 (local address 127.31.227.3:46241)
0812 00:12:05.450005 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:05.450046 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:05.450986 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:05.451265 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:05.451269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:05.451581 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:05.451785 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:05.451791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.452459 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.452460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:05.453298 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:05.453301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.453711 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.453712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:05.453819 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.454299 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:05.454308 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:05.455790 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:05.457479 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:05.457488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:05.457491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:05.457759 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:05.457983 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:05.457984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:05.457985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:05.458103 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:05.458385 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:05.458393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:05.458540 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 00:12:05.458647 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 00:12:05.458817 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260812 00:12:05.459748 10978 heartbeater.cc:347] Connected to a master server at 127.31.227.62:33371
I20260812 00:12:05.459869 10978 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:05.460064 10978 heartbeater.cc:511] Master 127.31.227.62:33371 requested a full tablet report, sending...
I20260812 00:12:05.460471 10170 ts_manager.cc:194] Registered new tserver with Master: 62d97bf766d74a72a27441c077bb9efe (127.31.227.3:45663)
I20260812 00:12:05.460879 10170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:46241
I20260812 00:12:05.471109 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/test-admin.keytab.
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.488016 32652 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 00:12:05.489816 10988 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:05.488932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.31.227.1:33257)
0812 00:12:05.489110 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.489116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.489128 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:12:05.489167 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.489169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.489184 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.489274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.489722 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53038: BlockingRecv error: recv got EOF from 127.0.0.1:53038 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
W20260812 00:12:05.489948 32652 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.490978 32652 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.491933 32652 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:12:05.500253 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.492484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.31.227.1:33257)
0812 00:12:05.492538 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.492540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.492571 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:12:05.492638 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.492639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.492649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.492706 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.493108 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.493934 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.494666 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.494990 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.495047 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.497457 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:05.497467 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:05.497467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:05.497474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:05.499063 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.499406 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.499409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.499409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.499466 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.499635 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.499635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.499635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.499767 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:05.499888 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:05.500082 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:12:05.500164 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 00:12:05.502240 10988 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:05.501732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.31.227.1:33257)
0812 00:12:05.501769 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.501771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.501819 (+ 48us) server_negotiation.cc:407] Connection header received
0812 00:12:05.501869 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.501869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.501878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.501930 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.502171 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53042: BlockingRecv error: recv got EOF from 127.0.0.1:53042 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 00:12:05.502345 32652 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.503335 32652 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.504172 32652 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:12:05.511329 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.504711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.31.227.1:33257)
0812 00:12:05.504748 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.504751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.504846 (+ 95us) server_negotiation.cc:407] Connection header received
0812 00:12:05.504916 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.504917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.504933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.504995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.505373 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.505967 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.506688 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.506890 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.506995 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.508992 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:05.508993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:05.508994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:05.509000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:05.510327 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.510663 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.510665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.510665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.510722 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.510917 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.510918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.510919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.511018 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:05.511094 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:05.511198 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 00:12:05.511264 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 00:12:05.512823 10988 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:05.512378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.31.227.1:33257)
0812 00:12:05.512419 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.512421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.512429 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:05.512470 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.512470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.512477 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.512523 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.512786 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53068: BlockingRecv error: recv got EOF from 127.0.0.1:53068 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 00:12:05.512900 32652 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493496733958-32652-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.513855 32652 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:05.514710 32652 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[10093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:12:05.522097 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:05.515317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.31.227.1:33257)
0812 00:12:05.515372 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:05.515374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:05.515391 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:12:05.515450 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:05.515451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:05.515459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:05.515507 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:05.515869 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.516365 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.517006 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:05.517179 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:05.517229 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:05.519441 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:05.519442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:05.519443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:05.519450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:05.521050 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.521365 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.521366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.521366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.521419 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:05.521648 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:05.521649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:05.521650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:05.521744 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:05.521824 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:05.521905 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 00:12:05.521961 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:12:05.522779 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 10293
I20260812 00:12:05.538915 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 10522
I20260812 00:12:05.552439 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 10764
I20260812 00:12:05.572415 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 10117
2026-08-12T00:12:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2215 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[11014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[11014](info): set up 2 sockets
Aug 12 00:12:05 dist-test-slave-zr1t krb5kdc[11014](info): commencing operation
krb5kdc: starting...
W20260812 00:12:06.866627 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:06 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab.
Entry for principal oryx/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab.
I20260812 00:12:07.004549 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46125
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab
--principal=oryx/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:07.096648 11040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:07.096880 11040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:07.096920 11040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:07.098150 11040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:07.098213 11040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:07.098240 11040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:07.098271 11040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:07.098296 11040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:07.099772 11040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab
--principal=oryx/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:07.100052 11040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:07.100284 11040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:07.105235 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.105263 11053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.105310 11052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.105571 11040 server_base.cc:1034] running on GCE node
I20260812 00:12:07.105831 11040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:07.106174 11040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:07.107357 11040 hybrid_clock.cc:648] HybridClock initialized: now 1786493527107341 us; error 44 us; skew 500 ppm
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:07.110121 11040 init.cc:377] Logged in from keytab as oryx/127.31.227.62@KRBTEST.COM (short username oryx)
I20260812 00:12:07.110620 11040 webserver.cc:460] Webserver started at http://127.31.227.62:35521/ using document root <none> and password file <none>
I20260812 00:12:07.110868 11040 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.110947 11040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.111101 11040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:07.112211 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "a6170ba2510b40ff8145edab8b083fcb"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "0a491dd3ae29d919a0bc67de2a329c8c"
server_key_iv: "05228b31c39d52eb6846ea0ec0a90bc8"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.112610 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "a6170ba2510b40ff8145edab8b083fcb"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "0a491dd3ae29d919a0bc67de2a329c8c"
server_key_iv: "05228b31c39d52eb6846ea0ec0a90bc8"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.114054 11040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:07.114784 11066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.114959 11040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:07.115072 11040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "a6170ba2510b40ff8145edab8b083fcb"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "0a491dd3ae29d919a0bc67de2a329c8c"
server_key_iv: "05228b31c39d52eb6846ea0ec0a90bc8"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.115149 11040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.125048 11040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:07.127383 11040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:07.127560 11040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:07.132026 11040 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:46125
I20260812 00:12:07.132062 11154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:46125 every 8 connection(s)
I20260812 00:12:07.132457 11040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:07.133433 11155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:07.135721 11155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Bootstrap starting.
I20260812 00:12:07.136245 11155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:07.136482 11155 log.cc:824] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:07.137145 11155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: No bootstrap required, opened a new log
I20260812 00:12:07.138715 11155 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } }
I20260812 00:12:07.138851 11155 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:07.138873 11155 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6170ba2510b40ff8145edab8b083fcb, State: Initialized, Role: FOLLOWER
I20260812 00:12:07.139014 11155 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } }
I20260812 00:12:07.139101 11155 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:07.139123 11155 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:07.139153 11155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:07.140050 11155 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } }
I20260812 00:12:07.140178 11155 leader_election.cc:302] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6170ba2510b40ff8145edab8b083fcb; no voters:
I20260812 00:12:07.140213 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 11040
I20260812 00:12:07.140326 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:07.140357 11155 leader_election.cc:288] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:07.140457 11158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:07.140550 32652 external_mini_cluster.cc:1428] Setting key 206337f98403f3338a964df40018b6a6
I20260812 00:12:07.140645 11155 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:07.140669 11158 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 LEADER]: Becoming Leader. State: Replica: a6170ba2510b40ff8145edab8b083fcb, State: Running, Role: LEADER
I20260812 00:12:07.140755 11158 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } }
I20260812 00:12:07.141093 11159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } }
I20260812 00:12:07.141199 11159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:07.141567 11160 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6170ba2510b40ff8145edab8b083fcb. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } }
I20260812 00:12:07.141729 11160 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:07.141888 11175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:07.142755 11175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:07.144382 11175 catalog_manager.cc:1347] Generated new cluster ID: fb813337782d49bdab45c0a3717ea4e9
I20260812 00:12:07.144433 11175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:07.151187 11175 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:07.151855 11175 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:07.153095 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.141235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.31.227.62:46125)
0812 00:12:07.141597 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.141608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.141653 (+ 45us) server_negotiation.cc:407] Connection header received
0812 00:12:07.141782 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.141790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.141852 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.142868 (+ 1016us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.142978 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.144292 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.145494 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.146130 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.146228 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.149099 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.149103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.149104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.149118 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.151307 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.151710 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.151714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.151714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.151840 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.152090 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.152091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.152091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.152383 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.152544 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.152688 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.152852 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":230,"threads_started":1}
I20260812 00:12:07.156669 11175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Generated new TSK 0
I20260812 00:12:07.156873 11175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab.
Entry for principal oryx/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab.
I20260812 00:12:07.186339 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46125
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:07.272262 11201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:07.272493 11201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:07.272519 11201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:07.273728 11201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:07.273772 11201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:07.273799 11201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:07.275300 11201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:07.275561 11201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:07.275813 11201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:07.281036 11207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.281059 11211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.281057 11208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.281350 11201 server_base.cc:1034] running on GCE node
I20260812 00:12:07.281592 11201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:07.281821 11201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:07.282980 11201 hybrid_clock.cc:648] HybridClock initialized: now 1786493527282936 us; error 28 us; skew 500 ppm
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:07.285715 11201 init.cc:377] Logged in from keytab as oryx/127.31.227.1@KRBTEST.COM (short username oryx)
I20260812 00:12:07.286151 11201 webserver.cc:460] Webserver started at http://127.31.227.1:44029/ using document root <none> and password file <none>
I20260812 00:12:07.286396 11201 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.286468 11201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.286628 11201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:07.287684 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "9f5f406446794158bbd519f829730575"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c6c04e979b2ee9238b9b58cf57d2b785"
server_key_iv: "34f5b942c94955f6c59da1c835bf59da"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.288062 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "9f5f406446794158bbd519f829730575"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c6c04e979b2ee9238b9b58cf57d2b785"
server_key_iv: "34f5b942c94955f6c59da1c835bf59da"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.289422 11201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:12:07.290174 11220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.290354 11201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:07.290462 11201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "9f5f406446794158bbd519f829730575"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c6c04e979b2ee9238b9b58cf57d2b785"
server_key_iv: "34f5b942c94955f6c59da1c835bf59da"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.290549 11201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.296648 11201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:07.298453 11201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:07.298607 11201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:07.298839 11201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:07.299142 11201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:07.299198 11201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.299238 11201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:07.299265 11201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.305254 11201 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:34005
I20260812 00:12:07.305310 11413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:34005 every 8 connection(s)
I20260812 00:12:07.305696 11201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:07.312109 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 11201
I20260812 00:12:07.312204 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:07.312405 32652 external_mini_cluster.cc:1428] Setting key ecea64bdb104c309a1b172e57df89daf
I20260812 00:12:07.314735 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.306406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:44559 (local address 127.31.227.62:46125)
0812 00:12:07.306495 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.306497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.306654 (+ 157us) server_negotiation.cc:407] Connection header received
0812 00:12:07.307121 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.307122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.307132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.307182 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.308049 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.308542 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.309463 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.309717 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.309770 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.311745 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.311746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.311746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.311752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.313303 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.313811 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.313812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.313812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.313865 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.314136 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.314137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.314137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.314234 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.314286 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.314603 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.314670 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:12:07.315024 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.306368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:44559)
0812 00:12:07.306538 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.306573 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.307038 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.307234 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.307237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.307789 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.307971 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.307975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.308620 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.308620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.309389 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.309390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.309788 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.309789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.309890 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.310288 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.310296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.311655 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.313382 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.313390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.313392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.313738 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.313929 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.313929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.313930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.314073 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.314297 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.314303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.314468 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.314581 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.314750 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":117,"threads_started":1}
I20260812 00:12:07.315529 11416 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46125
I20260812 00:12:07.315618 11416 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:07.315809 11416 heartbeater.cc:511] Master 127.31.227.62:46125 requested a full tablet report, sending...
I20260812 00:12:07.316315 11100 ts_manager.cc:194] Registered new tserver with Master: 9f5f406446794158bbd519f829730575 (127.31.227.1:34005)
I20260812 00:12:07.316929 11100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:44559
WARNING: no policy specified for oryx/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab.
Entry for principal oryx/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab.
I20260812 00:12:07.336004 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46125
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:07.419426 11429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:07.419638 11429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:07.419661 11429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:07.420766 11429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:07.420804 11429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:07.420832 11429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:07.422427 11429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:07.422628 11429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:07.422814 11429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:07.427819 11443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.427807 11441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.427807 11440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.428023 11429 server_base.cc:1034] running on GCE node
I20260812 00:12:07.428303 11429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:07.428560 11429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:07.429708 11429 hybrid_clock.cc:648] HybridClock initialized: now 1786493527429683 us; error 35 us; skew 500 ppm
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:07.432296 11429 init.cc:377] Logged in from keytab as oryx/127.31.227.2@KRBTEST.COM (short username oryx)
I20260812 00:12:07.432744 11429 webserver.cc:460] Webserver started at http://127.31.227.2:44569/ using document root <none> and password file <none>
I20260812 00:12:07.432996 11429 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.433066 11429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.433218 11429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:07.434331 11429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "ff1d832ba2de433b9c890e5d935a4ca5"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "6265fda7ef393591fbf248f25d14a6b3"
server_key_iv: "308885b74adb22b0ee90e46d9a755e70"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.434695 11429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "ff1d832ba2de433b9c890e5d935a4ca5"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "6265fda7ef393591fbf248f25d14a6b3"
server_key_iv: "308885b74adb22b0ee90e46d9a755e70"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.436074 11429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:07.436823 11454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.437114 11429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:07.437222 11429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "ff1d832ba2de433b9c890e5d935a4ca5"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "6265fda7ef393591fbf248f25d14a6b3"
server_key_iv: "308885b74adb22b0ee90e46d9a755e70"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.437295 11429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.447746 11429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:07.449613 11429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:07.449787 11429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:07.450029 11429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:07.450356 11429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:07.450412 11429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.450448 11429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:07.450477 11429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.456728 11429 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:37507
I20260812 00:12:07.456785 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:37507 every 8 connection(s)
I20260812 00:12:07.457100 11429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:07.460901 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 11429
I20260812 00:12:07.461015 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:07.461256 32652 external_mini_cluster.cc:1428] Setting key 484fd78dc5131fbbd1d862d8773e8c99
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:07.467383 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.457855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46229 (local address 127.31.227.62:46125)
0812 00:12:07.457920 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.457922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.458145 (+ 223us) server_negotiation.cc:407] Connection header received
0812 00:12:07.458619 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.458620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.458630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.458696 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.459629 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.460123 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.461636 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.461899 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.461973 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.464465 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.464467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.464467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.464474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.465967 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.466397 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.466399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.466399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.466462 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.466722 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.466723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.466723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.466842 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.466920 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.467196 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.467277 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:12:07.467589 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.457842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:46229)
0812 00:12:07.458017 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.458059 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.458537 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.458750 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.458753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.459333 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.459552 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.459556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.460206 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.460207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.461522 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.461548 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.461995 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.461997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.462145 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.462736 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.462745 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.464351 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.466080 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.466088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.466091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.466317 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.466534 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.466535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.466535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.466651 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.466912 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.466919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.467063 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.467174 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.467335 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260812 00:12:07.468114 11641 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46125
I20260812 00:12:07.468222 11641 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:07.468428 11641 heartbeater.cc:511] Master 127.31.227.62:46125 requested a full tablet report, sending...
I20260812 00:12:07.468878 11100 ts_manager.cc:194] Registered new tserver with Master: ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507)
I20260812 00:12:07.469408 11100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:46229
WARNING: no policy specified for oryx/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab.
Entry for principal oryx/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab.
I20260812 00:12:07.488395 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46125
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:07.573369 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:07.573601 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:07.573626 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:07.574733 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:07.574767 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:07.574793 11648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:07.576189 11648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:07.576372 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:07.576550 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:07.581619 11656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.581631 11655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.581710 11648 server_base.cc:1034] running on GCE node
W20260812 00:12:07.581678 11659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.581998 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:07.582298 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:07.583518 11648 hybrid_clock.cc:648] HybridClock initialized: now 1786493527583501 us; error 26 us; skew 500 ppm
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:07.586133 11648 init.cc:377] Logged in from keytab as oryx/127.31.227.3@KRBTEST.COM (short username oryx)
I20260812 00:12:07.586638 11648 webserver.cc:460] Webserver started at http://127.31.227.3:44241/ using document root <none> and password file <none>
I20260812 00:12:07.586880 11648 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.586949 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.587105 11648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:07.588160 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "57f22567680d4bf8b570e0ca4fd1be1f"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "10ff41de1a8283845b5453b806d8f08a"
server_key_iv: "7f6b702d061d42e8e21cea7b1492c597"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.588559 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "57f22567680d4bf8b570e0ca4fd1be1f"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "10ff41de1a8283845b5453b806d8f08a"
server_key_iv: "7f6b702d061d42e8e21cea7b1492c597"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.589936 11648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:07.590657 11669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.590831 11648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:07.590924 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "57f22567680d4bf8b570e0ca4fd1be1f"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "10ff41de1a8283845b5453b806d8f08a"
server_key_iv: "7f6b702d061d42e8e21cea7b1492c597"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.591007 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.603971 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:07.606500 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:07.606662 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:07.606913 11648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:07.607223 11648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:07.607256 11648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.607300 11648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:07.607334 11648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.613665 11648 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:34555
I20260812 00:12:07.613729 11862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:34555 every 8 connection(s)
I20260812 00:12:07.614039 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:07.623657 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.614737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:50659 (local address 127.31.227.62:46125)
0812 00:12:07.614790 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.614792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.615087 (+ 295us) server_negotiation.cc:407] Connection header received
0812 00:12:07.615578 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.615579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.615590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.615644 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.616572 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.617063 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.618127 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.618332 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.618404 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.620670 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.620672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.620672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.620682 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.622007 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.622642 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.622643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.622644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.622713 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.622988 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.622989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.622989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.623109 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.623185 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.623475 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.623562 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:07.623879 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.614725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:50659)
0812 00:12:07.614905 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.614973 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.615486 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.615699 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.615702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.616281 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.616489 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.616494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.617146 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.617146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.618026 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.618028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.618420 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.618421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.618523 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.619090 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.619099 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.620556 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.622278 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.622299 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.622302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.622554 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.622790 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.622791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.622791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.622911 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.623187 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.623195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.623339 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.623453 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.623617 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260812 00:12:07.624235 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 11648
I20260812 00:12:07.624382 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:07.624424 11864 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46125
I20260812 00:12:07.624526 11864 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:07.624634 32652 external_mini_cluster.cc:1428] Setting key 3ad56bf430a8a9ae717e79922cf2daa0
I20260812 00:12:07.624739 11864 heartbeater.cc:511] Master 127.31.227.62:46125 requested a full tablet report, sending...
I20260812 00:12:07.625216 11100 ts_manager.cc:194] Registered new tserver with Master: 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:07.625761 11100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:50659
I20260812 00:12:07.625785 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.227.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.61.keytab.
Entry for principal oryx/127.31.227.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.61.keytab.
I20260812 00:12:07.652258 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:44461
--webserver_interface=127.31.227.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.61.keytab
--principal=oryx/127.31.227.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46125,127.31.227.61:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:07.737447 11873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:07.737718 11873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:07.737758 11873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:07.738912 11873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:07.738983 11873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:07.739012 11873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:07.739044 11873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:07.739069 11873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:07.740814 11873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46125,127.31.227.61:44461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.61.keytab
--principal=oryx/127.31.227.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.227.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
W20260812 00:12:07.740949 11873 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.227.62:46125,127.31.227.61:44461'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 00:12:07.741132 11873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:07.741422 11873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:07.746439 11882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.746431 11883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.746572 11873 server_base.cc:1034] running on GCE node
W20260812 00:12:07.746430 11886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.746863 11873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:07.747182 11873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:07.748358 11873 hybrid_clock.cc:648] HybridClock initialized: now 1786493527748336 us; error 38 us; skew 500 ppm
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:07.751035 11873 init.cc:377] Logged in from keytab as oryx/127.31.227.61@KRBTEST.COM (short username oryx)
I20260812 00:12:07.751502 11873 webserver.cc:460] Webserver started at http://127.31.227.61:38733/ using document root <none> and password file <none>
I20260812 00:12:07.751703 11873 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.751747 11873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.751839 11873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:07.752872 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/instance:
uuid: "e8fe10ed12464d32b4234a170523cfaa"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c710a9745c27792a406cda9e394e1d5b"
server_key_iv: "85af589aa356bf5ea7b82b042cd97a33"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.753186 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal/instance:
uuid: "e8fe10ed12464d32b4234a170523cfaa"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c710a9745c27792a406cda9e394e1d5b"
server_key_iv: "85af589aa356bf5ea7b82b042cd97a33"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.754583 11873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:12:07.755276 11899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.755488 11873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:07.755581 11873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
uuid: "e8fe10ed12464d32b4234a170523cfaa"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c710a9745c27792a406cda9e394e1d5b"
server_key_iv: "85af589aa356bf5ea7b82b042cd97a33"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.755638 11873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.768738 11873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:07.770634 11873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:07.770744 11873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:07.774704 11873 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.61:44461
I20260812 00:12:07.774761 11994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.61:44461 every 8 connection(s)
I20260812 00:12:07.775123 11873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
I20260812 00:12:07.776146 11997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:07.777958 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 11873
I20260812 00:12:07.777776 11997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.61@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
I20260812 00:12:07.787853 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.778591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.31.227.62:46125)
0812 00:12:07.778644 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.778647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.778916 (+ 269us) server_negotiation.cc:407] Connection header received
0812 00:12:07.779489 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.779490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.779505 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.779591 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.780143 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.780635 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.781703 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.781970 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.782021 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.784288 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.784289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.784289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.784295 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.786125 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.786616 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.786618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.786618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.786681 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.787035 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.787035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.787036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.787225 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.787330 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.787657 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.787744 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:12:07.788079 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.778561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:60522)
0812 00:12:07.778761 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.778801 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.779321 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.779618 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.779620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.779860 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.780019 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.780023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.780733 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.780734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.781627 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.781630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.782043 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.782044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.782157 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.782713 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.782721 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.784203 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.786211 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.786218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.786220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.786529 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.786763 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.786764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.786764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.786962 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.787338 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.787347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.787528 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.787610 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.787772 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260812 00:12:07.788316 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.778572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.31.227.61:44461)
0812 00:12:07.778791 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.778801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.778832 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:12:07.779315 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.779324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.779349 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.779444 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.780119 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.781073 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.781866 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.782249 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.782340 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.784878 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.784886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.784887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.784894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.786695 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.787184 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.787187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.787188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.787262 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.787473 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.787474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.787474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.787632 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.787794 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.787877 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.788205 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"spinlock_wait_cycles":8960,"thread_start_us":113,"threads_started":1}
I20260812 00:12:07.788805 11997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.61@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
I20260812 00:12:07.795182 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.789084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:60446)
0812 00:12:07.789136 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.789146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.789190 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.789292 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.789293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.789402 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.789439 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.789441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.790083 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.790084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.790563 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.790565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.790882 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.790883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.790934 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.791326 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.791331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.792568 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.794204 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.794205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.794205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.794356 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.794577 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.794578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.794578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.794654 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.794850 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.794851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.794888 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.794984 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.795078 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 00:12:07.795231 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.789130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.31.227.61:44461)
0812 00:12:07.789179 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.789181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.789189 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:07.789240 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.789240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.789248 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.789290 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.789478 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.790016 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.790633 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.790818 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.790871 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.792646 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.792647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.792648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.792653 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.794133 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.794433 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.794433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.794434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.794481 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.794696 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.794697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.794697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.794783 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.794857 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.795007 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.795125 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":31}
I20260812 00:12:07.796550 11997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Bootstrap starting.
I20260812 00:12:07.797402 11997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:07.797739 11997 log.cc:824] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:07.798476 11997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: No bootstrap required, opened a new log
I20260812 00:12:07.799887 11997 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } }
I20260812 00:12:07.800024 11997 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:07.800047 11997 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8fe10ed12464d32b4234a170523cfaa, State: Initialized, Role: FOLLOWER
I20260812 00:12:07.800204 11997 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } }
I20260812 00:12:07.800500 12005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } }
I20260812 00:12:07.800549 11997 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:07.800596 12005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:12:07.802367 12020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:07.802431 12020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:07.804183 11873 master_runner.cc:428] Detected that this master e8fe10ed12464d32b4234a170523cfaa is joining an existing cluster
I20260812 00:12:07.804275 11873 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.227.61:44461
I20260812 00:12:07.812597 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.805450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.31.227.62:46125)
0812 00:12:07.805502 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.805509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.806048 (+ 539us) server_negotiation.cc:407] Connection header received
0812 00:12:07.806128 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.806129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.806141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.806198 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.806601 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.807325 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.808430 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.808693 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.808784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.809835 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.809837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.809837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.809845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.811366 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.811788 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.811789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.811790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.811851 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.812024 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.812025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.812025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.812162 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.812245 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.812421 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.812497 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:07.812704 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.805366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:60532)
0812 00:12:07.805994 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.806003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.806070 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.806221 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.806222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.806451 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.806513 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.806515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.807410 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.807411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.808353 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.808356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.808813 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.808814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.808899 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.809271 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.809276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.809750 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.811480 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.811481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.811482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.811710 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.811908 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.811909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.811909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.811971 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.812236 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.812237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.812294 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.812385 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.812511 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":30,"thread_start_us":104,"threads_started":1}
I20260812 00:12:07.813251 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.805575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.31.227.61:44461)
0812 00:12:07.805632 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.805634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.805900 (+ 266us) server_negotiation.cc:407] Connection header received
0812 00:12:07.805979 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.805979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.805988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.806054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.806519 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.807259 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.808068 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.808262 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.808334 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.809511 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.809513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.809513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.809555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.811416 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.811862 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.811862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.811863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.811914 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.812087 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.812088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.812089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.812208 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.812267 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.812435 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.812617 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":37}
I20260812 00:12:07.813766 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.805366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:60460)
0812 00:12:07.805825 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.805837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.805907 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.806059 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.806060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.806321 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.806435 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.806438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.807344 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.807345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.807982 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.807984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.808359 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.808361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.808434 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.808954 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.808960 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.809420 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.811505 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.811506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.811506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.811788 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.811973 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.811974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.811974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.812030 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.812292 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.812292 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.812334 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.812462 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.812568 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.62@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
I20260812 00:12:07.824107 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.815145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:60466)
0812 00:12:07.815315 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.815363 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.815445 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.815598 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.815602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.815978 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.816079 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.816082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.817324 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.817325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.818413 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.818417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.818914 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.818915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.819006 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.819458 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.819464 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.820992 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.822838 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.822845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.822847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.823063 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.823295 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.823296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.823297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.823448 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.823762 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.823770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.823820 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.823915 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.824020 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260812 00:12:07.824139 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.815257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.31.227.61:44461)
0812 00:12:07.815343 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.815345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.815417 (+ 72us) server_negotiation.cc:407] Connection header received
0812 00:12:07.815526 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.815527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.815535 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.815578 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.816492 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.817180 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.818539 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.818831 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.818893 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.821086 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.821087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.821088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.821097 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.822752 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.823137 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.823139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.823139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.823202 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.823523 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.823524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.823525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.823664 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.823738 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.823964 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.824043 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:12:07.825014 11100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.227.61:44461: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa"
member_type: NON_VOTER
last_known_addr {
host: "127.31.227.61"
port: 44461
}
attrs {
promote: true
}
}
dest_uuid: "a6170ba2510b40ff8145edab8b083fcb"
cas_config_opid_index: -1
I20260812 00:12:07.825366 11100 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } }
I20260812 00:12:07.825989 12032 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e8fe10ed12464d32b4234a170523cfaa (127.31.227.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:07.826392 11949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:07.827143 12032 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.227.61:44461
I20260812 00:12:07.827229 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:07.827302 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:07.827453 11949 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Refusing update from remote peer a6170ba2510b40ff8145edab8b083fcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 00:12:07.827756 11873 master.cc:537] Master@127.31.227.61:44461 shutting down...
I20260812 00:12:07.827811 12029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:12:07.828538 11070 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 00:12:07.828972 11070 consensus_peers.cc:487] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb -> Peer e8fe10ed12464d32b4234a170523cfaa (127.31.227.61:44461): Couldn't send request to peer e8fe10ed12464d32b4234a170523cfaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:12:07.830880 11873 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:07.830989 11873 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:07.831043 11873 tablet_replica.cc:330] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: stopping tablet replica
I20260812 00:12:07.844107 11873 master.cc:552] Master@127.31.227.61:44461 shutdown complete.
I20260812 00:12:07.844161 11873 master_runner.cc:305] Clearing existing system tablet
I20260812 00:12:07.844309 11873 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.844363 11873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.845044 11873 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:07.845571 12040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.845693 11873 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.845752 11873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.845775 11873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
uuid: "e8fe10ed12464d32b4234a170523cfaa"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c710a9745c27792a406cda9e394e1d5b"
server_key_iv: "85af589aa356bf5ea7b82b042cd97a33"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.846505 11873 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:07.847862 11873 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 00:12:07.847923 11873 log.cc:1196] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 00:12:07.848137 11873 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Deleting consensus metadata
I20260812 00:12:07.848270 11873 master_runner.cc:315] Copying system tablet from 127.31.227.62:46125
I20260812 00:12:07.848934 11873 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.62:46125
I20260812 00:12:07.855459 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.849343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.31.227.62:46125)
0812 00:12:07.849393 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.849396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.849459 (+ 63us) server_negotiation.cc:407] Connection header received
0812 00:12:07.849602 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.849603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.849614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.849676 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.850074 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.850574 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.851652 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.851837 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.851886 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.852950 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.852951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.852952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.852958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.854385 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.854786 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.854786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.854787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.854839 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.855040 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.855041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.855041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.855138 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.855188 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.855339 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.855396 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:12:07.855547 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.849242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:60540)
0812 00:12:07.849390 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:07.849403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:07.849478 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:07.849698 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:07.849699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:07.849915 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:07.849996 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.849998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.850653 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.850654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.851564 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:07.851567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.851909 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.851910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:07.851993 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.852457 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:07.852462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:07.852860 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:07.854468 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.854469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.854469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.854705 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.854901 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:07.854902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:07.854902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:07.854978 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:07.855198 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:07.855199 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:07.855248 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:12:07.855324 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:12:07.855434 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260812 00:12:07.856209 11125 tablet_copy_service.cc:140] P a6170ba2510b40ff8145edab8b083fcb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e8fe10ed12464d32b4234a170523cfaa ({username='oryx', principal='oryx/127.31.227.61@KRBTEST.COM'} at 127.0.0.1:60540)
I20260812 00:12:07.856318 11125 tablet_copy_service.cc:161] P a6170ba2510b40ff8145edab8b083fcb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e8fe10ed12464d32b4234a170523cfaa at {username='oryx', principal='oryx/127.31.227.61@KRBTEST.COM'} at 127.0.0.1:60540: session id = e8fe10ed12464d32b4234a170523cfaa-00000000000000000000000000000000
I20260812 00:12:07.857175 11125 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 00:12:07.857832 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 00:12:07.858788 11873 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: tablet copy: Starting download of 0 data blocks...
I20260812 00:12:07.858974 11873 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: tablet copy: Starting download of 1 WAL segments...
I20260812 00:12:07.859853 11873 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 00:12:07.860649 11125 tablet_copy_service.cc:342] P a6170ba2510b40ff8145edab8b083fcb: Request end of tablet copy session e8fe10ed12464d32b4234a170523cfaa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.227.61@KRBTEST.COM'} at 127.0.0.1:60540
I20260812 00:12:07.860728 11125 tablet_copy_service.cc:433] P a6170ba2510b40ff8145edab8b083fcb: ending tablet copy session e8fe10ed12464d32b4234a170523cfaa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e8fe10ed12464d32b4234a170523cfaa
I20260812 00:12:07.863344 11873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:07.864723 12054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:07.864706 12057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.864897 11873 server_base.cc:1034] running on GCE node
W20260812 00:12:07.864697 12059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:07.865103 11873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:07.865248 11873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:07.866380 11873 hybrid_clock.cc:648] HybridClock initialized: now 1786493527866375 us; error 40 us; skew 500 ppm
Aug 12 00:12:07 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:07.867719 11873 init.cc:377] Logged in from keytab as oryx/127.31.227.61@KRBTEST.COM (short username oryx)
I20260812 00:12:07.868026 11873 webserver.cc:460] Webserver started at http://127.31.227.61:40489/ using document root <none> and password file <none>
I20260812 00:12:07.868170 11873 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:07.868218 11873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:07.868852 11873 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:07.869419 12073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.869581 11873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:07.869632 11873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
uuid: "e8fe10ed12464d32b4234a170523cfaa"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c710a9745c27792a406cda9e394e1d5b"
server_key_iv: "85af589aa356bf5ea7b82b042cd97a33"
server_key_version: "encryptionkey@0"
I20260812 00:12:07.869693 11873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:07.887481 11873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:07.889154 11873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:07.892562 11873 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.61:44461
I20260812 00:12:07.892781 12160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.61:44461 every 8 connection(s)
I20260812 00:12:07.892952 11873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
I20260812 00:12:07.893898 12161 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:12:07.894399 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Bootstrap starting.
I20260812 00:12:07.896684 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:07.896922 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: Bootstrap complete.
I20260812 00:12:07.897138 12161 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } }
I20260812 00:12:07.897230 12161 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8fe10ed12464d32b4234a170523cfaa, State: Initialized, Role: LEARNER
I20260812 00:12:07.897284 12161 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } }
I20260812 00:12:07.897465 12161 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:07.897487 12164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:07.897619 12164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: LEARNER
I20260812 00:12:07.898962 12183 catalog_manager.cc:1259] Loaded cluster ID: fb813337782d49bdab45c0a3717ea4e9
I20260812 00:12:07.899009 12183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: loading cluster ID for follower catalog manager: success
I20260812 00:12:07.900656 12183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: acquiring CA information for follower catalog manager: success
I20260812 00:12:07.900991 12183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:12:07.947844 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:07.940438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.31.227.61:44461)
0812 00:12:07.940596 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:07.940598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:07.940610 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:12:07.940645 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:07.940646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:07.940659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:07.940751 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:07.941457 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.942569 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.943319 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:07.943662 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:07.943739 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:07.945004 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:07.945006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:07.945007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:07.945014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:07.946719 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.947067 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.947068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.947068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.947137 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:07.947336 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:07.947337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:07.947337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:07.947476 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:07.947574 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:07.947660 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 00:12:07.947745 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.31.227.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.60.keytab.
Entry for principal oryx/127.31.227.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.60.keytab.
I20260812 00:12:07.976356 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:40935
--webserver_interface=127.31.227.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.60.keytab
--principal=oryx/127.31.227.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:08.067158 12190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:08.067358 12190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:08.067378 12190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:08.068571 12190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:08.068612 12190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:08.068627 12190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:08.068639 12190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:08.068686 12190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:08.070344 12190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.60.keytab
--principal=oryx/127.31.227.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.227.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:08.070609 12190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:08.070896 12190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:08.076022 12198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.075959 12196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.076061 12190 server_base.cc:1034] running on GCE node
W20260812 00:12:08.075973 12195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.076527 12190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:08.076818 12190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:08.077980 12190 hybrid_clock.cc:648] HybridClock initialized: now 1786493528077957 us; error 37 us; skew 500 ppm
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:08.080644 12190 init.cc:377] Logged in from keytab as oryx/127.31.227.60@KRBTEST.COM (short username oryx)
I20260812 00:12:08.081069 12190 webserver.cc:460] Webserver started at http://127.31.227.60:36117/ using document root <none> and password file <none>
I20260812 00:12:08.081307 12190 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:08.081413 12190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:08.081601 12190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:08.082764 12190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/instance:
uuid: "a9f27d8833fb4991acd38c9713ccca64"
format_stamp: "Formatted at 2026-08-12 00:12:08 on dist-test-slave-zr1t"
server_key: "e18c1159c89f94863d1c51b20cd458d6"
server_key_iv: "cc698db4b6e2877841793284408b82e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.083135 12190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal/instance:
uuid: "a9f27d8833fb4991acd38c9713ccca64"
format_stamp: "Formatted at 2026-08-12 00:12:08 on dist-test-slave-zr1t"
server_key: "e18c1159c89f94863d1c51b20cd458d6"
server_key_iv: "cc698db4b6e2877841793284408b82e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.084470 12190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:08.085186 12211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.085392 12190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:08.085506 12190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
uuid: "a9f27d8833fb4991acd38c9713ccca64"
format_stamp: "Formatted at 2026-08-12 00:12:08 on dist-test-slave-zr1t"
server_key: "e18c1159c89f94863d1c51b20cd458d6"
server_key_iv: "cc698db4b6e2877841793284408b82e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.085633 12190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:08.110110 12190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:08.112171 12190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:08.112365 12190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:08.116638 12190 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.60:40935
I20260812 00:12:08.116675 12291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.60:40935 every 8 connection(s)
I20260812 00:12:08.117076 12190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
I20260812 00:12:08.118455 12292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:08.120167 12292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:12:08.122588 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 12190
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.60@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.227.60@KRBTEST.COM
I20260812 00:12:08.130775 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.121022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.31.227.62:46125)
0812 00:12:08.121075 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.121077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.121177 (+ 100us) server_negotiation.cc:407] Connection header received
0812 00:12:08.121706 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.121707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.121717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.121783 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.122402 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.123017 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.124511 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.124821 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.124912 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.127564 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.127566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.127566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.127581 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.129123 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.129675 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.129677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.129677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.129744 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.130048 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.130049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.130049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.130204 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.130278 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.130572 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.130661 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:12:08.130802 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.120859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:60544)
0812 00:12:08.121044 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.121087 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.121618 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.121856 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.121859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.122121 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.122327 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.122331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.123138 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.123140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.124421 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.124424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.124965 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.124966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.125078 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.125699 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.125710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.127443 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.129211 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.129217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.129219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.129596 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.129824 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.129825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.129825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.129966 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.130278 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.130284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.130415 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.130500 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.130620 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260812 00:12:08.131434 12292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:12:08.132510 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.123105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.31.227.60:40935)
0812 00:12:08.123267 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.123272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.123291 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:12:08.123320 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.123321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.123337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.123400 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.124168 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.125144 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.126205 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.126596 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.126668 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.129006 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.129015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.129015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.129023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.131198 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.131550 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.131554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.131554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.131617 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.131857 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.131858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.131859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.132037 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.132224 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.132326 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.132404 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.60@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
I20260812 00:12:08.138186 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.131726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:60516)
0812 00:12:08.131789 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.131800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.131839 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.132101 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.132102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.132261 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.132303 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.132305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.133092 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.133093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.133699 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.133701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.134082 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.134083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.134144 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.134538 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.134542 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.135681 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.137331 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.137331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.137332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.137479 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.137723 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.137724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.137724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.137775 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.137963 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.137964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.138002 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.138056 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.138127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 00:12:08.138216 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.131926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.31.227.61:44461)
0812 00:12:08.131979 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.131981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.131987 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:12:08.132010 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.132011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.132023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.132084 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.132415 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.132970 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.133786 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.134005 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.134084 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.135766 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.135768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.135768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.135775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.137259 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.137608 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.137609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.137609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.137663 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.137813 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.137813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.137813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.137913 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.137960 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.138075 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.138153 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:08.138667 12292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.60@KRBTEST.COM for oryx/127.31.227.60@KRBTEST.COM
I20260812 00:12:08.145045 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.138939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.0.0.1:60218)
0812 00:12:08.138989 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.139000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.139047 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.139169 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.139169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.139298 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.139323 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.139324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.140128 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.140129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.140747 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.140749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.141134 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.141136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.141184 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.141567 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.141572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.142775 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.144073 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.144074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.144074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.144235 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.144436 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.144437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.144437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.144501 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.144733 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.144734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.144771 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.144837 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.144929 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 00:12:08.145097 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.138999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.31.227.60:40935)
0812 00:12:08.139029 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.139031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.139039 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:08.139102 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.139103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.139112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.139166 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.139393 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.140031 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.140834 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.141053 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.141129 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.142858 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.142860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.142860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.142866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.143968 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.144306 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.144307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.144308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.144362 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.144562 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.144563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.144563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.144667 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.144741 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.144864 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.144978 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":14}
I20260812 00:12:08.146687 12292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Bootstrap starting.
I20260812 00:12:08.147567 12292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:08.147867 12292 log.cc:824] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:08.148669 12292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: No bootstrap required, opened a new log
I20260812 00:12:08.150265 12292 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } }
I20260812 00:12:08.150427 12292 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:08.150480 12292 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f27d8833fb4991acd38c9713ccca64, State: Initialized, Role: FOLLOWER
I20260812 00:12:08.150635 12292 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } }
I20260812 00:12:08.150949 12292 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:08.150920 12298 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } } }
I20260812 00:12:08.151051 12298 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:12:08.152645 12313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:08.152701 12313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:08.155273 12190 master_runner.cc:428] Detected that this master a9f27d8833fb4991acd38c9713ccca64 is joining an existing cluster
I20260812 00:12:08.155339 12190 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.227.60:40935
I20260812 00:12:08.163868 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.156582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.31.227.62:46125)
0812 00:12:08.156694 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.156697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.156982 (+ 285us) server_negotiation.cc:407] Connection header received
0812 00:12:08.157010 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.157011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.157026 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.157078 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.157619 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.158408 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.159603 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.159913 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.159956 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.161125 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.161126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.161126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.161133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.162594 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.163033 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.163035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.163035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.163103 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.163326 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.163327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.163328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.163447 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.163524 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.163695 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.163774 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 00:12:08.163890 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.156513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:60554)
0812 00:12:08.156878 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.156886 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.156941 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.157128 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.157129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.157387 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.157479 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.157481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.158512 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.158513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.159497 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.159501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.160038 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.160039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.160105 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.160533 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.160539 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.161010 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.162689 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.162690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.162691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.162941 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.163202 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.163202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.163203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.163257 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.163524 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.163525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.163589 (+ 64us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.163669 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.163775 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":38,"thread_start_us":87,"threads_started":1}
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493527, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.62@KRBTEST.COM for oryx/127.31.227.60@KRBTEST.COM
I20260812 00:12:08.174093 12298 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:08.174180 12298 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } }
I20260812 00:12:08.174474 12298 leader_election.cc:288] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), e8fe10ed12464d32b4234a170523cfaa (127.31.227.61:44461)
I20260812 00:12:08.174675 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f27d8833fb4991acd38c9713ccca64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6170ba2510b40ff8145edab8b083fcb" is_pre_election: true
I20260812 00:12:08.174690 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f27d8833fb4991acd38c9713ccca64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8fe10ed12464d32b4234a170523cfaa" is_pre_election: true
I20260812 00:12:08.174804 12122 raft_consensus.cc:1799] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Handling vote request from an unknown peer a9f27d8833fb4991acd38c9713ccca64
I20260812 00:12:08.174858 12122 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate a9f27d8833fb4991acd38c9713ccca64 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260812 00:12:08.175105 12217 leader_election.cc:302] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9f27d8833fb4991acd38c9713ccca64; no voters: a6170ba2510b40ff8145edab8b083fcb, e8fe10ed12464d32b4234a170523cfaa
I20260812 00:12:08.175225 12298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:08.184479 12298 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 00:12:08.185252 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 11201
I20260812 00:12:08.189627 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.165987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.0.0.1:60234)
0812 00:12:08.166045 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.166057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.166104 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.166371 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.166372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.166541 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.166586 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.166587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.167214 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.167215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.168039 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.168041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.168381 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.168382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.168452 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.168793 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.168805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.170254 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.188502 (+ 18248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.188503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.188504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.188727 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.188918 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.188918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.188919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.188972 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.189248 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.189249 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.189294 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.189339 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.189455 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:12:08.190366 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.166075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.31.227.60:40935)
0812 00:12:08.166222 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.166224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.166230 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:12:08.166271 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.166272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.166292 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.166341 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.166671 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.167140 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.168120 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.168312 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.168359 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.170352 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.170354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.170354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.170361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.188400 (+ 18039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.188807 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.188808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.188808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.188857 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.189030 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.189030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.189030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.189175 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.190100 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.190141 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.190230 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260812 00:12:08.217327 11100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.227.60:40935: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64"
member_type: NON_VOTER
last_known_addr {
host: "127.31.227.60"
port: 40935
}
attrs {
promote: true
}
}
dest_uuid: "a6170ba2510b40ff8145edab8b083fcb"
cas_config_opid_index: 5
I20260812 00:12:08.217741 11100 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } }
I20260812 00:12:08.218814 12032 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.220214 12032 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.227.60:40935
I20260812 00:12:08.220315 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.220415 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.221513 12257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Refusing update from remote peer a6170ba2510b40ff8145edab8b083fcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 00:12:08.228911 12029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:08.231055 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.156758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.0.0.1:60226)
0812 00:12:08.157083 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.157092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.157149 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.157290 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.157291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.157578 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.157629 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.157631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.158498 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.158498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.159503 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.159506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.160004 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.160005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.160077 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.182269 (+ 22192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.182275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.183193 (+ 918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.185105 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.185106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.185107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.216839 (+ 31732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.217963 (+ 1124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.217965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.217966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.218054 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.230601 (+ 12547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.230603 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.230685 (+ 82us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.230764 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.230927 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":72,"thread_start_us":71,"threads_started":1}
I20260812 00:12:08.231616 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.156785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.31.227.60:40935)
0812 00:12:08.156827 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.156828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.157130 (+ 302us) server_negotiation.cc:407] Connection header received
0812 00:12:08.157208 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.157209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.157217 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.157242 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.157702 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.158412 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.159596 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.159869 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.159906 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.183306 (+ 23400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.183308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.183308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.183314 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.184975 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.216972 (+ 31997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.216974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.216975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.217068 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.218136 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.218137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.218137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.218282 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.218345 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.231437 (+ 13092us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.231529 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:12:08.232120 12190 master.cc:537] Master@127.31.227.60:40935 shutting down...
I20260812 00:12:08.232862 12257 raft_consensus.cc:858] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } }
W20260812 00:12:08.232954 12257 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } }
I20260812 00:12:08.233839 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.156628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.31.227.61:44461)
0812 00:12:08.156672 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.156674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.156824 (+ 150us) server_negotiation.cc:407] Connection header received
0812 00:12:08.157052 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.157053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.157065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.157120 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.157676 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.158416 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.159597 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.159892 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.159947 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.182972 (+ 23025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.182975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.182976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.182988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.185101 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.232486 (+ 47385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.232488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.232489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.232587 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.232815 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.232816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.232817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.232961 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.233023 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.233621 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.233722 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:12:08.233542 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.156513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:60532)
0812 00:12:08.156758 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.156770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.156977 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.157173 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.157174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.157497 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.157584 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.157586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.158512 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.158513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.159503 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.159506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.160005 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.160005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.160066 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.182088 (+ 22022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.182094 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.182612 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.185191 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.185192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.185192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.185476 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.232664 (+ 47188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.232665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.232665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.232730 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.233175 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.233177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.233225 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.233282 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.233399 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":65,"thread_start_us":92,"threads_started":1}
I20260812 00:12:08.233541 12257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e8fe10ed12464d32b4234a170523cfaa (127.31.227.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:08.238629 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.220746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:60548)
0812 00:12:08.234589 (+ 13843us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.234602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.234659 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.234900 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.234901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.234903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.234956 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.234958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.235936 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.235938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.237239 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.237243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.238350 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.238352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.238428 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.238455 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.238511 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.238556 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13820}
I20260812 00:12:08.238777 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:34005
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:46125
--webserver_port=44029
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:08.239228 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.221873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.31.227.61:44461)
0812 00:12:08.234153 (+ 12280us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.234156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.234715 (+ 559us) server_negotiation.cc:407] Connection header received
0812 00:12:08.234740 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.234741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.234767 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.234833 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.235027 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.235792 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.237351 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.238031 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.238105 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.238177 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.239118 (+ 941us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.239161 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12263,"thread_start_us":818,"threads_started":1}
I20260812 00:12:08.239487 12337 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:08.239584 12337 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 00:12:08.239621 12337 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:08.239701 12122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Refusing update from remote peer a6170ba2510b40ff8145edab8b083fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 00:12:08.239717 12337 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 00:12:08.240043 12298 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6170ba2510b40ff8145edab8b083fcb. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } }
I20260812 00:12:08.240123 12298 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 00:12:08.240429 12335 mvcc.cc:205] Tried to move back new op lower bound from 7317477487232286720 to 7317477487168954368. Current Snapshot: MvccSnapshot[applied={T|T < 7317477487180320768}]
I20260812 00:12:08.241035 12029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 00:12:08.241147 12029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.241206 12029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.241993 12122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
W20260812 00:12:08.242626 11070 consensus_peers.cc:487] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb -> Peer a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60:40935): Couldn't send request to peer a9f27d8833fb4991acd38c9713ccca64. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:12:08.243116 12190 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260812 00:12:08.243240 12190 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260812 00:12:08.243289 12190 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: stopping tablet replica
I20260812 00:12:08.243363 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.243412 12331 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: attempting to promote NON_VOTER e8fe10ed12464d32b4234a170523cfaa to VOTER
I20260812 00:12:08.243435 12164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6170ba2510b40ff8145edab8b083fcb. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: NON_VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: true } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.243538 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: LEARNER
I20260812 00:12:08.243558 12164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: LEARNER
I20260812 00:12:08.243664 12331 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } }
W20260812 00:12:08.244354 11070 consensus_peers.cc:487] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb -> Peer a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60:40935): Couldn't send request to peer a9f27d8833fb4991acd38c9713ccca64. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:12:08.244431 12122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 LEARNER]: Refusing update from remote peer a6170ba2510b40ff8145edab8b083fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 00:12:08.244623 12029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 00:12:08.245460 12029 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e8fe10ed12464d32b4234a170523cfaa (127.31.227.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.246469 12029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.246537 12029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.246583 12029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.246624 12029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.246961 12122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e8fe10ed12464d32b4234a170523cfaa (127.31.227.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.247933 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.248054 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:08.257414 12190 master.cc:552] Master@127.31.227.60:40935 shutdown complete.
I20260812 00:12:08.257478 12190 master_runner.cc:305] Clearing existing system tablet
I20260812 00:12:08.257799 12190 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:08.257856 12190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:08.258726 12190 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:08.259322 12349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.259470 12190 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:08.259541 12190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:08.259565 12190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
uuid: "a9f27d8833fb4991acd38c9713ccca64"
format_stamp: "Formatted at 2026-08-12 00:12:08 on dist-test-slave-zr1t"
server_key: "e18c1159c89f94863d1c51b20cd458d6"
server_key_iv: "cc698db4b6e2877841793284408b82e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.260296 12190 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:08.261680 12190 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 00:12:08.261754 12190 log.cc:1196] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 00:12:08.261984 12190 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Deleting consensus metadata
I20260812 00:12:08.262105 12190 master_runner.cc:315] Copying system tablet from 127.31.227.62:46125
I20260812 00:12:08.262753 12190 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.62:46125
I20260812 00:12:08.270346 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.263142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.31.227.62:46125)
0812 00:12:08.263198 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.263200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.263255 (+ 55us) server_negotiation.cc:407] Connection header received
0812 00:12:08.263379 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.263380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.263394 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.263470 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.263872 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.264620 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.265765 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.266073 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.266158 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.267312 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.267313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.267314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.267322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.269035 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.269442 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.269443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.269444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.269512 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.269778 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.269779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.269780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.269922 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.269997 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.270176 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.270256 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:12:08.270686 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.263073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:60564)
0812 00:12:08.263179 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.263192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.263286 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.263499 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.263500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.263713 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.263771 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.263774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.264723 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.264725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.265635 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.265639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.266165 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.266166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.266226 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.266783 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.266788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.267183 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.269135 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.269137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.269137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.269362 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.269624 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.269626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.269626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.269700 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.270010 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.270011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.270064 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.270145 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.270308 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260812 00:12:08.271060 11125 tablet_copy_service.cc:140] P a6170ba2510b40ff8145edab8b083fcb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a9f27d8833fb4991acd38c9713ccca64 ({username='oryx', principal='oryx/127.31.227.60@KRBTEST.COM'} at 127.0.0.1:60564)
I20260812 00:12:08.271118 11125 tablet_copy_service.cc:161] P a6170ba2510b40ff8145edab8b083fcb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a9f27d8833fb4991acd38c9713ccca64 at {username='oryx', principal='oryx/127.31.227.60@KRBTEST.COM'} at 127.0.0.1:60564: session id = a9f27d8833fb4991acd38c9713ccca64-00000000000000000000000000000000
I20260812 00:12:08.271708 11125 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 00:12:08.272205 12190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 00:12:08.273501 12190 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: tablet copy: Starting download of 0 data blocks...
I20260812 00:12:08.273729 12190 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: tablet copy: Starting download of 1 WAL segments...
I20260812 00:12:08.274516 12190 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 00:12:08.275441 11125 tablet_copy_service.cc:342] P a6170ba2510b40ff8145edab8b083fcb: Request end of tablet copy session a9f27d8833fb4991acd38c9713ccca64-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.227.60@KRBTEST.COM'} at 127.0.0.1:60564
I20260812 00:12:08.275497 11125 tablet_copy_service.cc:433] P a6170ba2510b40ff8145edab8b083fcb: ending tablet copy session a9f27d8833fb4991acd38c9713ccca64-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a9f27d8833fb4991acd38c9713ccca64
I20260812 00:12:08.278322 12190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:08.279551 12361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.279709 12362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.280014 12366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.280207 12190 server_base.cc:1034] running on GCE node
I20260812 00:12:08.280395 12190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:08.280535 12190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:08.281634 12190 hybrid_clock.cc:648] HybridClock initialized: now 1786493528281624 us; error 28 us; skew 500 ppm
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:08.283313 12190 init.cc:377] Logged in from keytab as oryx/127.31.227.60@KRBTEST.COM (short username oryx)
I20260812 00:12:08.283638 12190 webserver.cc:460] Webserver started at http://127.31.227.60:44819/ using document root <none> and password file <none>
I20260812 00:12:08.283812 12190 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:08.283885 12190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:08.284679 12190 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:08.285275 12380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.285394 12190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.285434 12190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
uuid: "a9f27d8833fb4991acd38c9713ccca64"
format_stamp: "Formatted at 2026-08-12 00:12:08 on dist-test-slave-zr1t"
server_key: "e18c1159c89f94863d1c51b20cd458d6"
server_key_iv: "cc698db4b6e2877841793284408b82e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.285538 12190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:08.302719 12190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:08.305336 12190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:08.308733 12190 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.60:40935
I20260812 00:12:08.308830 12469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.60:40935 every 8 connection(s)
I20260812 00:12:08.309095 12190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
I20260812 00:12:08.310083 12470 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:12:08.310636 12470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Bootstrap starting.
I20260812 00:12:08.312997 12470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:08.313213 12470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Bootstrap complete.
I20260812 00:12:08.313401 12470 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } }
I20260812 00:12:08.313505 12470 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9f27d8833fb4991acd38c9713ccca64, State: Initialized, Role: LEARNER
I20260812 00:12:08.313576 12470 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } }
I20260812 00:12:08.313759 12470 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:08.313768 12473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.314009 12473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:12:08.315236 12506 catalog_manager.cc:1259] Loaded cluster ID: fb813337782d49bdab45c0a3717ea4e9
I20260812 00:12:08.315857 12506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: loading cluster ID for follower catalog manager: success
I20260812 00:12:08.316939 12506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: acquiring CA information for follower catalog manager: success
I20260812 00:12:08.317269 12506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 00:12:08.343438 12340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:08.343662 12340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:08.343701 12340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:08.344812 12340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:08.344877 12340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:08.344921 12340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:08.346518 12340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:34005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=44029
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:08.346783 12340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:08.347024 12340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:08.352262 12513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.352382 12516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.352391 12340 server_base.cc:1034] running on GCE node
W20260812 00:12:08.352254 12514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.353101 12340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:08.353699 12340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:08.354907 12340 hybrid_clock.cc:648] HybridClock initialized: now 1786493528354897 us; error 60 us; skew 500 ppm
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:08.357725 12340 init.cc:377] Logged in from keytab as oryx/127.31.227.1@KRBTEST.COM (short username oryx)
I20260812 00:12:08.358204 12340 webserver.cc:460] Webserver started at http://127.31.227.1:44029/ using document root <none> and password file <none>
I20260812 00:12:08.358420 12340 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:08.358474 12340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:08.359774 12340 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:08.360348 12540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.360574 12340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:08.360671 12340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "9f5f406446794158bbd519f829730575"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "c6c04e979b2ee9238b9b58cf57d2b785"
server_key_iv: "34f5b942c94955f6c59da1c835bf59da"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.361016 12340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:08.369081 12340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:08.371073 12340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:08.371212 12340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:08.371382 12340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:08.371635 12340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:08.371666 12340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.371687 12340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:08.371701 12340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.377664 12340 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:34005
I20260812 00:12:08.377746 12728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:34005 every 8 connection(s)
I20260812 00:12:08.378013 12340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.60@KRBTEST.COM
I20260812 00:12:08.387068 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 12340
I20260812 00:12:08.387236 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 11429
I20260812 00:12:08.389287 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.378880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:33983 (local address 127.31.227.62:46125)
0812 00:12:08.378931 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.378933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.379081 (+ 148us) server_negotiation.cc:407] Connection header received
0812 00:12:08.379720 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.379721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.379736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.379785 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.380723 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.381225 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382632 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.382927 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382987 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.386079 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.386081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.386081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.386090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.387806 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.388252 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.388253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.388254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.388306 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.388595 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.388596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.388596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.388725 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.388763 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.389076 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.389157 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:08.389856 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.378790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:33983)
0812 00:12:08.378990 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.379002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.379608 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.379858 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.379871 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.380417 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.380620 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.380625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.381337 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.381338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.382542 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.382545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.383041 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.383042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.383119 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.383780 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.383790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.385955 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.387891 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.387898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.387900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.388179 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.388415 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.388415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.388416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.388533 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.388800 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.388804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.388953 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.389391 (+ 438us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.389594 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260812 00:12:08.392643 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.379191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:60685 (local address 127.31.227.61:44461)
0812 00:12:08.379250 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.379253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.379313 (+ 60us) server_negotiation.cc:407] Connection header received
0812 00:12:08.379728 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.379730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.379743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.379777 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.380732 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.381482 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382642 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.382850 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382887 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.386932 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.386934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.386934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.386942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.391085 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.391553 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.391555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.391555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.391623 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.391879 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.391880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.391881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.392075 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.392131 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.392392 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.392492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:12:08.393080 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.379017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.31.227.1:60685)
0812 00:12:08.379262 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.379271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.379621 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.379858 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.379862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.380432 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.380621 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.380628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.381609 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.381610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.382543 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.382545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.382984 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.383116 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.383917 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.383922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.386820 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.391190 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.391192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.391192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.391452 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.391710 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.391711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.391712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.391785 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.392190 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.392191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.392244 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.392757 (+ 513us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.392893 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":8,"thread_start_us":69,"threads_started":1}
I20260812 00:12:08.402877 12733 heartbeater.cc:347] Connected to a master server at 127.31.227.61:44461
I20260812 00:12:08.403020 12733 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.403239 12733 heartbeater.cc:511] Master 127.31.227.61:44461 requested a full tablet report, sending...
I20260812 00:12:08.403858 12106 ts_manager.cc:194] Registered new tserver with Master: 9f5f406446794158bbd519f829730575 (127.31.227.1:34005)
I20260812 00:12:08.406243 12731 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46125
I20260812 00:12:08.406307 12731 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.406387 12731 heartbeater.cc:511] Master 127.31.227.62:46125 requested a full tablet report, sending...
I20260812 00:12:08.406690 11100 ts_manager.cc:194] Re-registered known tserver with Master: 9f5f406446794158bbd519f829730575 (127.31.227.1:34005)
I20260812 00:12:08.406715 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.378725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.31.227.1:51949)
0812 00:12:08.378943 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.378995 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.379609 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.379887 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.379887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.380440 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.380621 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.380628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.381823 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.381825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.382457 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.382459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.382974 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.383116 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.383837 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.383841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.386586 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.405386 (+ 18800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.405387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.405388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.405690 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.405953 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.405954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.405955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.406038 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.406376 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.406378 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.406425 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.406497 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.406620 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
I20260812 00:12:08.407080 11100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:33983
I20260812 00:12:08.407189 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.378735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:51949 (local address 127.31.227.60:40935)
0812 00:12:08.379034 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.379037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.379206 (+ 169us) server_negotiation.cc:407] Connection header received
0812 00:12:08.379768 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.379769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.379781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.379844 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.380732 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.381729 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382564 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.382880 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.382949 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.386683 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.386684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.386685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.386693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.405269 (+ 18576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.405773 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.405774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.405775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.405870 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.406101 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.406102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.406102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.406272 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.406344 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.406541 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.406666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":75,"threads_started":1}
I20260812 00:12:08.407760 12729 heartbeater.cc:347] Connected to a master server at 127.31.227.60:40935
I20260812 00:12:08.407816 12729 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.407876 12729 heartbeater.cc:511] Master 127.31.227.60:40935 requested a full tablet report, sending...
I20260812 00:12:08.408241 12407 ts_manager.cc:194] Registered new tserver with Master: 9f5f406446794158bbd519f829730575 (127.31.227.1:34005)
I20260812 00:12:08.410835 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:37507
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:46125
--webserver_port=44569
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:08.496026 12743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:08.496230 12743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:08.496266 12743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:08.497542 12743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:08.497583 12743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:08.497633 12743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:08.499253 12743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:37507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=44569
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:08.499503 12743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:08.499740 12743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:08.504825 12756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.504868 12743 server_base.cc:1034] running on GCE node
W20260812 00:12:08.504846 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.504846 12754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.505247 12743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:08.505501 12743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:08.506706 12743 hybrid_clock.cc:648] HybridClock initialized: now 1786493528506696 us; error 28 us; skew 500 ppm
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:08.513557 12743 init.cc:377] Logged in from keytab as oryx/127.31.227.2@KRBTEST.COM (short username oryx)
I20260812 00:12:08.514084 12743 webserver.cc:460] Webserver started at http://127.31.227.2:44569/ using document root <none> and password file <none>
I20260812 00:12:08.514338 12743 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:08.514392 12743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:08.515936 12743 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:08.516530 12772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.516728 12743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:08.516822 12743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "ff1d832ba2de433b9c890e5d935a4ca5"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "6265fda7ef393591fbf248f25d14a6b3"
server_key_iv: "308885b74adb22b0ee90e46d9a755e70"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.517169 12743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:08.533159 12743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:08.535526 12743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:08.535701 12743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:08.535965 12743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:08.536298 12743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:08.536355 12743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.536405 12743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:08.536446 12743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.542850 12743 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:37507
I20260812 00:12:08.542937 12965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:37507 every 8 connection(s)
I20260812 00:12:08.543324 12743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:08.546420 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 12743
I20260812 00:12:08.546587 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 11648
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:08.564587 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.544308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:52729 (local address 127.31.227.62:46125)
0812 00:12:08.544360 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.544362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.544478 (+ 116us) server_negotiation.cc:407] Connection header received
0812 00:12:08.545103 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.545104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.545118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.545151 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.546012 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.546782 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.557701 (+ 10919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.557985 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.558061 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.561239 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.561241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.561242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.561251 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.563063 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.563537 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.563538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.563538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.563591 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.563837 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.563838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.563839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.563987 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.564027 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.564367 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.564453 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:08.565188 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.544125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:52729)
0812 00:12:08.544409 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.544419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.545001 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.545240 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.545244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.545708 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.545880 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.545894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.556957 (+ 11063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.556958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.557595 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.557600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.558126 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.558127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.558192 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.558719 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.558725 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.560831 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.563156 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.563166 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.563169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.563463 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.563651 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.563651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.563652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.563776 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.564068 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.564073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.564238 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.564725 (+ 487us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.564919 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":76,"thread_start_us":83,"threads_started":1}
I20260812 00:12:08.565618 12967 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46125
I20260812 00:12:08.565703 12967 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.565899 12967 heartbeater.cc:511] Master 127.31.227.62:46125 requested a full tablet report, sending...
I20260812 00:12:08.566231 11100 ts_manager.cc:194] Re-registered known tserver with Master: ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507)
I20260812 00:12:08.566665 11100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:52729
I20260812 00:12:08.578070 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.544659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:52609 (local address 127.31.227.61:44461)
0812 00:12:08.544726 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.544728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.544739 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:08.545104 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.545105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.545118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.545151 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.546005 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.546793 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.548148 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.548421 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.548457 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.560607 (+ 12150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.560609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.560610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.560620 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.576635 (+ 16015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.577138 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.577139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.577140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.577215 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.577423 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.577424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.577424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.577643 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.577716 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.577884 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.577972 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:12:08.578085 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.544451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.31.227.2:52609)
0812 00:12:08.544569 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.544579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.545000 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.545244 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.545244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.545793 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.545886 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.545887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.546924 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.546924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.548056 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.548058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.548535 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.548536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.548661 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.557813 (+ 9152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.557826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.560491 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.576735 (+ 16244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.576736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.576737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.577035 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.577300 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.577300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.577301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.577367 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.577717 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.577719 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.577766 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.577855 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.577970 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.60@KRBTEST.COM
I20260812 00:12:08.579909 12968 heartbeater.cc:347] Connected to a master server at 127.31.227.61:44461
I20260812 00:12:08.579982 12968 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.580088 12968 heartbeater.cc:511] Master 127.31.227.61:44461 requested a full tablet report, sending...
I20260812 00:12:08.580475 12106 ts_manager.cc:194] Registered new tserver with Master: ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507)
I20260812 00:12:08.581811 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:34555
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:46125
--webserver_port=44241
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:08.582741 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.544142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34389 (local address 127.31.227.60:40935)
0812 00:12:08.544195 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.544198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.544461 (+ 263us) server_negotiation.cc:407] Connection header received
0812 00:12:08.545097 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.545099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.545116 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.545167 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.546017 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.546766 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.576649 (+ 29883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.577000 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.577082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.579657 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.579659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.579659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.579668 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.581352 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.581801 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.581803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.581804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.581895 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.582148 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.582150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.582150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.582276 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.582379 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.582540 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.582632 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:08.582933 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.544125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.31.227.2:34389)
0812 00:12:08.544322 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.544366 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.544980 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.545238 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.545242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.545663 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.545879 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.545884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.575734 (+ 29850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.575736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.576523 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.576526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.577145 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.577146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.577222 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.578313 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.578319 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.579562 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.581447 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.581448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.581448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.581685 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.581997 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.581998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.581999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.582071 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.582372 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.582373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.582425 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.582496 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.582621 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260812 00:12:08.583384 12966 heartbeater.cc:347] Connected to a master server at 127.31.227.60:40935
I20260812 00:12:08.583446 12966 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.583498 12966 heartbeater.cc:511] Master 127.31.227.60:40935 requested a full tablet report, sending...
I20260812 00:12:08.583850 12407 ts_manager.cc:194] Registered new tserver with Master: ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507)
W20260812 00:12:08.666982 12974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:08.667191 12974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:08.667215 12974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:08.668365 12974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:08.668404 12974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:08.668454 12974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:08.670064 12974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:34555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=44241
--tserver_master_addrs=127.31.227.62:46125,127.31.227.61:44461,127.31.227.60:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:08.670315 12974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:08.670571 12974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:08.675841 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.675812 12988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:08.675869 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:08.676064 12974 server_base.cc:1034] running on GCE node
I20260812 00:12:08.676275 12974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:08.676553 12974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:08.677726 12974 hybrid_clock.cc:648] HybridClock initialized: now 1786493528677697 us; error 41 us; skew 500 ppm
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:08.680282 12974 init.cc:377] Logged in from keytab as oryx/127.31.227.3@KRBTEST.COM (short username oryx)
I20260812 00:12:08.680712 12974 webserver.cc:460] Webserver started at http://127.31.227.3:44241/ using document root <none> and password file <none>
I20260812 00:12:08.680968 12974 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:08.681058 12974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:08.682521 12974 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:08.683113 12999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.683279 12974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:08.683394 12974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "57f22567680d4bf8b570e0ca4fd1be1f"
format_stamp: "Formatted at 2026-08-12 00:12:07 on dist-test-slave-zr1t"
server_key: "10ff41de1a8283845b5453b806d8f08a"
server_key_iv: "7f6b702d061d42e8e21cea7b1492c597"
server_key_version: "encryptionkey@0"
I20260812 00:12:08.683745 12974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:08.711248 12974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:08.713382 12974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:08.713636 12974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:08.713909 12974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:08.714311 12974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:08.714372 12974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.714416 12974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:08.714443 12974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:08.721410 12974 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:34555
I20260812 00:12:08.721469 13186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:34555 every 8 connection(s)
I20260812 00:12:08.721840 12974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:08.728024 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 12974
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.60@KRBTEST.COM
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.61@KRBTEST.COM
Aug 12 00:12:08 dist-test-slave-zr1t krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493528, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:08.735658 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.722879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58943 (local address 127.31.227.60:40935)
0812 00:12:08.722927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.722929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.722939 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:08.724084 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.724086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.724100 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.724158 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.725187 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.725907 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727399 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.727622 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727714 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.732263 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.732264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.732265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.732274 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.734004 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.734499 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.734500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.734500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.734562 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.734864 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.734865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.734865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.735003 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.735048 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.735468 (+ 420us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.735554 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:08.736357 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.722616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.31.227.3:58943)
0812 00:12:08.722803 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.722843 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.723966 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.724241 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.724244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.724720 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.725067 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.725074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.726057 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.726058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.727296 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.727298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.727726 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.727858 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.728412 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.728426 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.732153 (+ 3727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.734096 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.734105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.734108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.734409 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.734641 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.734642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.734642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.734775 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.735101 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.735108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.735302 (+ 194us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.735905 (+ 603us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.736113 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":57,"thread_start_us":114,"threads_started":1}
I20260812 00:12:08.736950 13188 heartbeater.cc:347] Connected to a master server at 127.31.227.60:40935
I20260812 00:12:08.737044 13188 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.737229 13188 heartbeater.cc:511] Master 127.31.227.60:40935 requested a full tablet report, sending...
I20260812 00:12:08.737736 12407 ts_manager.cc:194] Registered new tserver with Master: 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:08.737893 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.722616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:48621)
0812 00:12:08.722798 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.722842 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.723960 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.724238 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.724243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.724721 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.725067 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.725074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.726110 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.726110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.727298 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.727301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.729066 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.729067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.729138 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.729757 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.729762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.731773 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.736674 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.736675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.736675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.736885 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.737113 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.737114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.737115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.737183 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.737507 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.737508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.737571 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.737647 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.737760 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":182,"thread_start_us":99,"threads_started":1}
I20260812 00:12:08.738183 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.722634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:48621 (local address 127.31.227.62:46125)
0812 00:12:08.722709 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.722712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.722921 (+ 209us) server_negotiation.cc:407] Connection header received
0812 00:12:08.724097 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.724098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.724112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.724149 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.725186 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.726007 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727408 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.727749 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727790 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.732027 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.732028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.732029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.732037 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.736278 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.736966 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.736967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.736967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.737039 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.737257 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.737258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.737258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.737396 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.737461 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.738000 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.738090 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:12:08.738195 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.722925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.31.227.3:46789)
0812 00:12:08.723120 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.723130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.723964 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.724244 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.724245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.724727 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:08.725067 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.725074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.726155 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.726156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.727298 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.727301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.727833 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.727904 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.728762 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:08.728768 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:08.731132 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:08.735940 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.735942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.735942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.736186 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.736507 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:08.736508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:08.736509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:08.736570 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:08.737876 (+ 1306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:08.737877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:08.737917 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.738007 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.738101 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":109,"thread_start_us":70,"threads_started":1}
I20260812 00:12:08.738602 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.723127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:46789 (local address 127.31.227.61:44461)
0812 00:12:08.723199 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.723201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.723210 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:08.724098 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.724099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.724121 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.724154 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.725186 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.726047 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727408 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.727705 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.727745 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.731232 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.731234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.731234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.731242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.735845 (+ 4603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.736363 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.736364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.736365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.736423 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.736639 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.736640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.736641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.737782 (+ 1141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.737848 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.738373 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.738487 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:12:08.738781 13189 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46125
I20260812 00:12:08.738852 13189 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.738917 13189 heartbeater.cc:511] Master 127.31.227.62:46125 requested a full tablet report, sending...
I20260812 00:12:08.738931 13191 heartbeater.cc:347] Connected to a master server at 127.31.227.61:44461
I20260812 00:12:08.738981 13191 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:08.739077 13191 heartbeater.cc:511] Master 127.31.227.61:44461 requested a full tablet report, sending...
I20260812 00:12:08.739145 11100 ts_manager.cc:194] Re-registered known tserver with Master: 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:08.739387 12106 ts_manager.cc:194] Registered new tserver with Master: 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:08.739602 11100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:48621
I20260812 00:12:08.741014 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.729662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.31.227.61:44461)
0812 00:12:08.731331 (+ 1669us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.731333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.731342 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:08.731364 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.731365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.731377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.731439 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.732838 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.733619 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.734651 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.734957 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.734997 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.736613 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.736614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.736615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.736622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.739364 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.739741 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.739742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.739743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.739813 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.740062 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.740063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.740063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.740202 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.740303 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.740517 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.740921 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641}
I20260812 00:12:08.741219 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.729593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.31.227.62:46125)
0812 00:12:08.730355 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.730357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.730364 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:08.730385 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.730386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.730398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.730471 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.731175 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.731931 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.733113 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.733474 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.733827 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.735044 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.735045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.735045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.735053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.739745 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.740099 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.740099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.740100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.740159 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.740371 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.740372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.740372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.740517 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.740983 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.741036 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.741125 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":86,"threads_started":1}
I20260812 00:12:08.742879 11100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:08.744491 11100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:08.744671 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.730020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.31.227.60:40935)
0812 00:12:08.730130 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.730132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.730262 (+ 130us) server_negotiation.cc:407] Connection header received
0812 00:12:08.730293 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.730294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.730310 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.730388 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.731249 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.736116 (+ 4867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.737346 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.738319 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.738404 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.741853 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.741854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.741855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.741862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.743288 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.743689 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.743690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.743690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.743748 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.744060 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.744061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.744061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.744202 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.744420 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.744461 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.744550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260812 00:12:08.752394 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.748244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:37507 (local address 127.0.0.1:45684)
0812 00:12:08.748520 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.748531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.748580 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.748930 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.748931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.748933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.748988 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.748991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.750332 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.750333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.751390 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.751392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.752142 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.752143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.752203 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.752209 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.752274 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.752318 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260812 00:12:08.752408 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.748502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.31.227.2:37507)
0812 00:12:08.748716 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.748722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.748733 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:08.748779 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.748782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.748800 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.749071 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.749142 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.750242 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.751475 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.752052 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.752141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.752221 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.752289 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.752333 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260812 00:12:08.752655 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.748194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34555 (local address 127.0.0.1:35696)
0812 00:12:08.748412 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.748422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.748479 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.749331 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.749331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.749334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.749391 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.749393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.750592 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.750593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.751525 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.751528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.752390 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.752391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.752474 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.752479 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.752527 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.752576 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":17,"thread_start_us":71,"threads_started":1}
I20260812 00:12:08.752918 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.748769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.31.227.3:34555)
0812 00:12:08.749168 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.749173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.749183 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:08.749217 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.749219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.749233 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.749315 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.749473 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.750506 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.751612 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.752300 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.752445 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.752675 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.752787 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.752842 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":86,"threads_started":1}
I20260812 00:12:08.752929 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.748194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:34005 (local address 127.0.0.1:52836)
0812 00:12:08.748313 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.748343 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.748395 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.748908 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.748909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.748911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.748960 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.748962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.750119 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.750122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.751306 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.751308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.752377 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.752378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.752780 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.752784 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.752835 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.752868 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260812 00:12:08.753165 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.748545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.31.227.1:34005)
0812 00:12:08.748753 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.748757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.748765 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:08.748798 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.748799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.748811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.748899 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.749030 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.750023 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.751409 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.752202 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.752273 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.752338 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.753044 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.753091 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260812 00:12:08.753333 12849 tablet_service.cc:1467] Processing CreateTablet for tablet 94cc238471ce4f09812ae5b58f262077 (DEFAULT_TABLE table=test-table [id=1220c8c06839412e9bd9946a610d53d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:08.753752 12849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cc238471ce4f09812ae5b58f262077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:08.753768 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 94cc238471ce4f09812ae5b58f262077 (DEFAULT_TABLE table=test-table [id=1220c8c06839412e9bd9946a610d53d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:08.754156 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cc238471ce4f09812ae5b58f262077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:08.756385 13237 tablet_bootstrap.cc:492] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Bootstrap starting.
I20260812 00:12:08.756703 13238 tablet_bootstrap.cc:492] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Bootstrap starting.
I20260812 00:12:08.757123 13237 tablet_bootstrap.cc:654] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:08.757354 13238 tablet_bootstrap.cc:654] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:08.757308 12626 tablet_service.cc:1467] Processing CreateTablet for tablet 94cc238471ce4f09812ae5b58f262077 (DEFAULT_TABLE table=test-table [id=1220c8c06839412e9bd9946a610d53d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:08.757485 13237 log.cc:824] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:08.757709 12626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cc238471ce4f09812ae5b58f262077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:08.757731 13238 log.cc:824] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:08.758460 13237 tablet_bootstrap.cc:492] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: No bootstrap required, opened a new log
I20260812 00:12:08.758515 13238 tablet_bootstrap.cc:492] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: No bootstrap required, opened a new log
I20260812 00:12:08.758550 13237 ts_tablet_manager.cc:1397] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:08.758565 13238 ts_tablet_manager.cc:1397] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:08.760560 13242 tablet_bootstrap.cc:492] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Bootstrap starting.
I20260812 00:12:08.761267 13242 tablet_bootstrap.cc:654] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:08.761161 13238 raft_consensus.cc:348] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.761343 13238 raft_consensus.cc:374] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:08.761260 13237 raft_consensus.cc:348] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.761390 13237 raft_consensus.cc:374] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:08.761390 13238 raft_consensus.cc:729] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f22567680d4bf8b570e0ca4fd1be1f, State: Initialized, Role: FOLLOWER
I20260812 00:12:08.761421 13237 raft_consensus.cc:729] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1d832ba2de433b9c890e5d935a4ca5, State: Initialized, Role: FOLLOWER
I20260812 00:12:08.761587 13238 consensus_queue.cc:260] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.761601 13237 consensus_queue.cc:260] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.761651 13242 log.cc:824] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:08.761878 13238 ts_tablet_manager.cc:1428] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:12:08.761914 13237 ts_tablet_manager.cc:1428] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:12:08.762013 12967 heartbeater.cc:503] Master 127.31.227.62:46125 was elected leader, sending a full tablet report...
I20260812 00:12:08.762440 13242 tablet_bootstrap.cc:492] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: No bootstrap required, opened a new log
I20260812 00:12:08.762512 13242 ts_tablet_manager.cc:1397] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:12:08.762629 13189 heartbeater.cc:503] Master 127.31.227.62:46125 was elected leader, sending a full tablet report...
I20260812 00:12:08.764683 13242 raft_consensus.cc:348] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.764853 13242 raft_consensus.cc:374] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:08.764894 13242 raft_consensus.cc:729] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5f406446794158bbd519f829730575, State: Initialized, Role: FOLLOWER
I20260812 00:12:08.765000 13242 consensus_queue.cc:260] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.765271 13242 ts_tablet_manager.cc:1428] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:12:08.765313 12731 heartbeater.cc:503] Master 127.31.227.62:46125 was elected leader, sending a full tablet report...
I20260812 00:12:08.767051 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.763174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.0.0.1:60242)
0812 00:12:08.763277 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.763289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.763329 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.763650 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.763651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.763653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.763698 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.763700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.764486 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.764487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.765632 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.765636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.766784 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.766786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.766859 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.766863 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.766915 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.766971 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 00:12:08.767345 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.763410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.31.227.60:40935)
0812 00:12:08.763476 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.763479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.763495 (+ 16us) server_negotiation.cc:407] Connection header received
0812 00:12:08.763520 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.763521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.763544 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.763603 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.763774 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.764404 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.765706 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.766309 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.766373 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.766445 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.767216 (+ 771us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.767264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:12:08.767568 12422 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260812 00:12:08.767989 12473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6170ba2510b40ff8145edab8b083fcb. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.768082 12473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: LEARNER
W20260812 00:12:08.794471 12969 tablet.cc:2369] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:08.853238 13244 raft_consensus.cc:482] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:08.853421 13244 raft_consensus.cc:504] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.853806 13244 leader_election.cc:288] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f5f406446794158bbd519f829730575 (127.31.227.1:34005), 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:08.856673 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.854041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:47707 (local address 127.31.227.3:34555)
0812 00:12:08.854093 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.854096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.854107 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:08.854167 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.854168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.854178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.854231 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.854415 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.854939 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.855906 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.856302 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.856393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.856402 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.856552 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.856597 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:12:08.856683 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.854130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46459 (local address 127.31.227.1:34005)
0812 00:12:08.854181 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.854183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.854191 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:08.854226 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.854227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.854237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.854294 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:08.854512 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.855000 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.855956 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.856306 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.856391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.856400 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.856582 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.856621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 00:12:08.856678 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.853977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34555 (local address 127.31.227.2:47707)
0812 00:12:08.854045 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.854057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.854105 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.854245 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.854246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.854248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.854329 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.854332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.855032 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.855033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.855812 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.855814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.856391 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.856392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.856475 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.856478 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.856523 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.856587 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":17}
I20260812 00:12:08.856698 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.853978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:34005 (local address 127.31.227.2:46459)
0812 00:12:08.854103 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:08.854115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:08.854164 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:08.854316 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:08.854317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:08.854319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:08.854416 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.854417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.855091 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.855092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.855839 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:08.855841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.856415 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.856416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:08.856490 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.856493 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:08.856562 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:12:08.856598 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":58}
I20260812 00:12:08.857050 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cc238471ce4f09812ae5b58f262077" candidate_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" is_pre_election: true
I20260812 00:12:08.857239 13114 raft_consensus.cc:2454] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1d832ba2de433b9c890e5d935a4ca5 in term 0.
I20260812 00:12:08.857533 12660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cc238471ce4f09812ae5b58f262077" candidate_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f5f406446794158bbd519f829730575" is_pre_election: true
I20260812 00:12:08.857559 12776 leader_election.cc:302] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f22567680d4bf8b570e0ca4fd1be1f, ff1d832ba2de433b9c890e5d935a4ca5; no voters:
I20260812 00:12:08.857647 12660 raft_consensus.cc:2454] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1d832ba2de433b9c890e5d935a4ca5 in term 0.
I20260812 00:12:08.857707 13244 raft_consensus.cc:2781] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:08.857760 13244 raft_consensus.cc:482] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:08.857790 13244 raft_consensus.cc:3037] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:08.858539 13244 raft_consensus.cc:504] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.858675 13244 leader_election.cc:288] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [CANDIDATE]: Term 1 election: Requested vote from peers 9f5f406446794158bbd519f829730575 (127.31.227.1:34005), 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:08.858839 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cc238471ce4f09812ae5b58f262077" candidate_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f22567680d4bf8b570e0ca4fd1be1f"
I20260812 00:12:08.858853 12660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cc238471ce4f09812ae5b58f262077" candidate_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f5f406446794158bbd519f829730575"
I20260812 00:12:08.858937 12660 raft_consensus.cc:3037] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:08.858942 13114 raft_consensus.cc:3037] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:08.859680 13114 raft_consensus.cc:2454] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1d832ba2de433b9c890e5d935a4ca5 in term 1.
I20260812 00:12:08.859756 12660 raft_consensus.cc:2454] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1d832ba2de433b9c890e5d935a4ca5 in term 1.
I20260812 00:12:08.859855 12776 leader_election.cc:302] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f22567680d4bf8b570e0ca4fd1be1f, ff1d832ba2de433b9c890e5d935a4ca5; no voters:
I20260812 00:12:08.859958 13244 raft_consensus.cc:2781] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:08.860138 13244 raft_consensus.cc:686] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 LEADER]: Becoming Leader. State: Replica: ff1d832ba2de433b9c890e5d935a4ca5, State: Running, Role: LEADER
I20260812 00:12:08.860249 13244 consensus_queue.cc:237] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } }
I20260812 00:12:08.861115 11100 catalog_manager.cc:5495] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2). New cstate: current_term: 1 leader_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } health_report { overall_health: HEALTHY } } }
I20260812 00:12:08.861799 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: NON_VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: true } } }
I20260812 00:12:08.861928 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.862635 12029 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: attempting to promote NON_VOTER a9f27d8833fb4991acd38c9713ccca64 to VOTER
I20260812 00:12:08.862764 12029 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:08.863344 12122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Refusing update from remote peer a6170ba2510b40ff8145edab8b083fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260812 00:12:08.863348 12422 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 LEARNER]: Refusing update from remote peer a6170ba2510b40ff8145edab8b083fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260812 00:12:08.863641 12032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260812 00:12:08.863785 12029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260812 00:12:08.864326 12331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.864569 12422 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.865375 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.865458 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.865540 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.865618 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.865556 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.865679 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:08.865801 12122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.865934 12473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.866016 12473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:08.866735 13260 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6170ba2510b40ff8145edab8b083fcb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:08.866832 13260 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:08.874704 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.872362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.31.227.2:37507)
0812 00:12:08.872426 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.872428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.872438 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:08.872487 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.872488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.872505 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.872568 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:08.872685 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.873154 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.873936 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.874128 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.874186 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.874365 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:08.874467 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:08.874533 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.874601 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.874622 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:12:08.876711 12660 raft_consensus.cc:1261] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Refusing update from remote peer ff1d832ba2de433b9c890e5d935a4ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:08.876711 13114 raft_consensus.cc:1261] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Refusing update from remote peer ff1d832ba2de433b9c890e5d935a4ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:08.877099 13256 consensus_queue.cc:1035] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:08.877254 13244 consensus_queue.cc:1035] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:12:08.878918 12734 tablet.cc:2369] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:08.880587 13265 mvcc.cc:205] Tried to move back new op lower bound from 7317477494275944448 to 7317477494212149248. Current Snapshot: MvccSnapshot[applied={T|T < 7317477494275944448}]
I20260812 00:12:08.882368 13268 mvcc.cc:205] Tried to move back new op lower bound from 7317477494275944448 to 7317477494212149248. Current Snapshot: MvccSnapshot[applied={T|T < 7317477494275944448}]
I20260812 00:12:08.884584 11100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:08.884682 11100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:08.886119 11100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6170ba2510b40ff8145edab8b083fcb: Sending DeleteTablet for 3 replicas of tablet 94cc238471ce4f09812ae5b58f262077
I20260812 00:12:08.886447 12849 tablet_service.cc:1514] Processing DeleteTablet for tablet 94cc238471ce4f09812ae5b58f262077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:08 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45684
I20260812 00:12:08.886518 12626 tablet_service.cc:1514] Processing DeleteTablet for tablet 94cc238471ce4f09812ae5b58f262077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:08 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:52836
I20260812 00:12:08.886688 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 94cc238471ce4f09812ae5b58f262077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:08 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:35696
I20260812 00:12:08.886783 13283 tablet_replica.cc:330] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: stopping tablet replica
I20260812 00:12:08.886885 13283 raft_consensus.cc:2229] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:08.887001 13283 raft_consensus.cc:2258] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:08.887179 13286 tablet_replica.cc:330] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: stopping tablet replica
I20260812 00:12:08.887280 13286 raft_consensus.cc:2229] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:08.887379 13286 raft_consensus.cc:2258] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:08.887694 13283 ts_tablet_manager.cc:1905] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:08.887768 13286 ts_tablet_manager.cc:1905] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:08.888002 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 11040
I20260812 00:12:08.899026 13286 ts_tablet_manager.cc:1918] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:08.899039 13283 ts_tablet_manager.cc:1918] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:08.899128 13283 log.cc:1196] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/94cc238471ce4f09812ae5b58f262077
I20260812 00:12:08.899128 13286 log.cc:1196] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/94cc238471ce4f09812ae5b58f262077
I20260812 00:12:08.899417 13286 ts_tablet_manager.cc:1939] T 94cc238471ce4f09812ae5b58f262077 P 57f22567680d4bf8b570e0ca4fd1be1f: Deleting consensus metadata
I20260812 00:12:08.899438 13283 ts_tablet_manager.cc:1939] T 94cc238471ce4f09812ae5b58f262077 P 9f5f406446794158bbd519f829730575: Deleting consensus metadata
I20260812 00:12:08.907390 13280 tablet_replica.cc:330] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: stopping tablet replica
I20260812 00:12:08.907555 13280 raft_consensus.cc:2229] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:08.907730 13280 raft_consensus.cc:2258] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:08.908272 13280 ts_tablet_manager.cc:1905] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:08.910478 13280 ts_tablet_manager.cc:1918] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:08.910563 13280 log.cc:1196] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/94cc238471ce4f09812ae5b58f262077
W20260812 00:12:08.910851 13003 connection.cc:535] server connection from 127.0.0.1:35696 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:12:08.910884 13280 ts_tablet_manager.cc:1939] T 94cc238471ce4f09812ae5b58f262077 P ff1d832ba2de433b9c890e5d935a4ca5: Deleting consensus metadata
W20260812 00:12:08.911182 12541 connection.cc:535] server connection from 127.0.0.1:52836 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 00:12:08.911250 12778 connection.cc:441] server connection from 127.0.0.1:45684 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45684 (request call id 1) could send its response
I20260812 00:12:08.915922 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.887674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.31.227.60:40935)
0812 00:12:08.887807 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.887810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.907242 (+ 19432us) server_negotiation.cc:407] Connection header received
0812 00:12:08.907290 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.907292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.907308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.907387 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.908256 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.909061 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.910187 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.910505 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.910551 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.912258 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.912259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.912259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.912268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.914293 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.914757 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.914758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.914758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.914820 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.915019 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.915019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.915020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.915143 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.915234 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.915739 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.915822 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 00:12:08.921761 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.913577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.31.227.61:44461)
0812 00:12:08.913677 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.913679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.913688 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:08.913716 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.913717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.913733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.913793 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.914574 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.915365 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.916617 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.916885 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.916945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.918475 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.918477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.918478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.918487 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.920257 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.920742 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.920743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.920743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.920812 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.921034 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.921036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.921036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.921171 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.921271 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.921540 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.921634 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:12:08.921818 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:08.913575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.31.227.60:40935)
0812 00:12:08.913628 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:08.913630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:08.913725 (+ 95us) server_negotiation.cc:407] Connection header received
0812 00:12:08.913828 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:08.913829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:08.913840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:08.913870 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:08.914976 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.915731 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.916812 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:08.917097 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:08.917170 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:08.918767 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:08.918768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:08.918769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:08.918776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:08.920508 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.920946 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.920947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.920947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.921004 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:08.921220 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:08.921221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:08.921222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:08.921353 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:08.921453 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:08.921632 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 00:12:08.921716 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:12:09.882479 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:09.882011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:35867)
0812 00:12:09.882323 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:09.882416 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":69,"threads_started":1}
I20260812 00:12:09.882781 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:09.882376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:59969)
0812 00:12:09.882643 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:09.882724 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260812 00:12:09.882856 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:09.882548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:51303)
0812 00:12:09.882740 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:09.882813 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":53,"threads_started":1}
W20260812 00:12:09.882969 12967 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:46125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
W20260812 00:12:09.882972 13189 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:46125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
W20260812 00:12:09.883185 12731 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:46125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:10.386930 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a6170ba2510b40ff8145edab8b083fcb)
I20260812 00:12:10.387058 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:10.387077 13303 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a6170ba2510b40ff8145edab8b083fcb)
I20260812 00:12:10.387255 13303 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:10.387353 13301 leader_election.cc:288] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), e8fe10ed12464d32b4234a170523cfaa (127.31.227.61:44461)
I20260812 00:12:10.387615 13303 leader_election.cc:288] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60:40935)
I20260812 00:12:10.387866 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.387508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52134)
0812 00:12:10.387733 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.387800 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":54,"threads_started":1}
I20260812 00:12:10.387885 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.387678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52146)
0812 00:12:10.387795 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.387845 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
W20260812 00:12:10.388046 12385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
W20260812 00:12:10.388336 12076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:10.388713 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.388606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52150)
0812 00:12:10.388649 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.388684 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 00:12:10.388810 12385 leader_election.cc:334] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:10.388965 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.388701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52164)
0812 00:12:10.388913 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.388939 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
W20260812 00:12:10.389050 12076 leader_election.cc:334] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:10.391000 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.387620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.31.227.61:44461)
0812 00:12:10.388097 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:10.388099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:10.388107 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:10.388139 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:10.388140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:10.388151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:10.388216 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:10.388399 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.389207 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.390258 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.390634 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.390717 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:10.390728 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:10.390870 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 00:12:10.390914 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":53,"threads_started":1}
I20260812 00:12:10.390995 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.387553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:44461 (local address 127.0.0.1:40524)
0812 00:12:10.387811 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.387823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:10.387888 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:10.388224 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:10.388225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:10.388227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:10.388291 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:10.388293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.389312 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.389313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:10.390153 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:10.390155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.390719 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.390720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:10.390789 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:10.390792 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:10.390861 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:12:10.390906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":68,"threads_started":1}
I20260812 00:12:10.391062 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.387989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.31.227.60:40935)
0812 00:12:10.388189 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:10.388191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:10.388199 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:10.388226 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:10.388227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:10.388238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:10.388469 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:10.388522 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.389260 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.390270 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.390651 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.390735 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:10.390747 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:10.390899 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 00:12:10.390945 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":38,"threads_started":1}
I20260812 00:12:10.391085 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.387802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:40935 (local address 127.0.0.1:43374)
0812 00:12:10.388017 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.388027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:10.388085 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:10.388338 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:10.388339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:10.388342 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:10.388396 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:10.388398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.389343 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.389344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:10.390165 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:10.390167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:10.390737 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:10.390738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:10.390803 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:10.390807 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:10.390891 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:12:10.390938 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":37,"threads_started":1}
I20260812 00:12:10.391415 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f27d8833fb4991acd38c9713ccca64" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8fe10ed12464d32b4234a170523cfaa" is_pre_election: true
I20260812 00:12:10.391525 12122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f27d8833fb4991acd38c9713ccca64 in term 1.
I20260812 00:12:10.391501 12422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8fe10ed12464d32b4234a170523cfaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9f27d8833fb4991acd38c9713ccca64" is_pre_election: true
I20260812 00:12:10.391629 12422 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8fe10ed12464d32b4234a170523cfaa in term 1.
I20260812 00:12:10.391722 12384 leader_election.cc:302] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9f27d8833fb4991acd38c9713ccca64, e8fe10ed12464d32b4234a170523cfaa; no voters: a6170ba2510b40ff8145edab8b083fcb
I20260812 00:12:10.391829 13301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 00:12:10.391894 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a6170ba2510b40ff8145edab8b083fcb)
I20260812 00:12:10.391884 12075 leader_election.cc:302] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9f27d8833fb4991acd38c9713ccca64, e8fe10ed12464d32b4234a170523cfaa; no voters: a6170ba2510b40ff8145edab8b083fcb
I20260812 00:12:10.391927 13301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:12:10.391990 13303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 00:12:10.392030 13303 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader a6170ba2510b40ff8145edab8b083fcb)
I20260812 00:12:10.392055 13303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:12:10.392874 13303 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:10.392880 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:10.393021 13301 leader_election.cc:288] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 2 election: Requested vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), e8fe10ed12464d32b4234a170523cfaa (127.31.227.61:44461)
I20260812 00:12:10.393019 13303 leader_election.cc:288] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [CANDIDATE]: Term 2 election: Requested vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), a9f27d8833fb4991acd38c9713ccca64 (127.31.227.60:40935)
I20260812 00:12:10.393218 12422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8fe10ed12464d32b4234a170523cfaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9f27d8833fb4991acd38c9713ccca64"
I20260812 00:12:10.393226 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f27d8833fb4991acd38c9713ccca64" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8fe10ed12464d32b4234a170523cfaa"
I20260812 00:12:10.393303 12422 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e8fe10ed12464d32b4234a170523cfaa in current term 2: Already voted for candidate a9f27d8833fb4991acd38c9713ccca64 in this term.
I20260812 00:12:10.393321 12122 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a9f27d8833fb4991acd38c9713ccca64 in current term 2: Already voted for candidate e8fe10ed12464d32b4234a170523cfaa in this term.
I20260812 00:12:10.393312 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.393222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52192)
0812 00:12:10.393260 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.393284 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 00:12:10.393270 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.393123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52180)
0812 00:12:10.393216 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.393248 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 00:12:10.393708 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.393597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52206)
0812 00:12:10.393655 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.393684 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 00:12:10.393795 12385 leader_election.cc:334] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:10.393850 12385 leader_election.cc:302] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9f27d8833fb4991acd38c9713ccca64; no voters: a6170ba2510b40ff8145edab8b083fcb, e8fe10ed12464d32b4234a170523cfaa
I20260812 00:12:10.393918 13301 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 00:12:10.394011 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.393747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52214)
0812 00:12:10.393947 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.393978 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260812 00:12:10.394068 12076 leader_election.cc:334] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:10.394105 12076 leader_election.cc:302] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8fe10ed12464d32b4234a170523cfaa; no voters: a6170ba2510b40ff8145edab8b083fcb, a9f27d8833fb4991acd38c9713ccca64
I20260812 00:12:10.394188 13303 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 00:12:10.883701 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.883352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:37165)
0812 00:12:10.883600 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.883643 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
I20260812 00:12:10.883772 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.883544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:34905)
0812 00:12:10.883682 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.883718 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260812 00:12:10.883847 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:10.883334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:38365)
0812 00:12:10.883770 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:10.883805 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":91,"threads_started":1}
I20260812 00:12:11.884518 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:11.884252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:36311)
0812 00:12:11.884404 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:11.884452 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260812 00:12:11.884653 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:11.884330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:45369)
0812 00:12:11.884565 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:11.884596 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":50,"threads_started":1}
I20260812 00:12:11.884677 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:11.884151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:38359)
0812 00:12:11.884481 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:11.884522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":65,"threads_started":1}
I20260812 00:12:12.372722 13326 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:12.372871 13326 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:12.373114 13326 leader_election.cc:288] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), e8fe10ed12464d32b4234a170523cfaa (127.31.227.61:44461)
I20260812 00:12:12.373395 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f27d8833fb4991acd38c9713ccca64" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8fe10ed12464d32b4234a170523cfaa" is_pre_election: true
I20260812 00:12:12.373473 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.373273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52468)
0812 00:12:12.373389 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.373431 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260812 00:12:12.373504 12122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f27d8833fb4991acd38c9713ccca64 in term 2.
I20260812 00:12:12.373708 12384 leader_election.cc:302] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f27d8833fb4991acd38c9713ccca64, e8fe10ed12464d32b4234a170523cfaa; no voters:
I20260812 00:12:12.373802 13326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 00:12:12.373848 13326 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:12.373883 13326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:12:12.374006 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.373908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52484)
0812 00:12:12.373944 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.373983 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260812 00:12:12.374111 12385 leader_election.cc:334] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:12.374739 13326 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:12.374884 13326 leader_election.cc:288] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 3 election: Requested vote from peers a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125), e8fe10ed12464d32b4234a170523cfaa (127.31.227.61:44461)
I20260812 00:12:12.375061 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.374954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52496)
0812 00:12:12.375007 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.375042 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:12:12.375056 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f27d8833fb4991acd38c9713ccca64" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8fe10ed12464d32b4234a170523cfaa"
I20260812 00:12:12.375149 12122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:12:12.375387 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.375328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52510)
0812 00:12:12.375352 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.375373 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260812 00:12:12.375494 12385 leader_election.cc:334] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
I20260812 00:12:12.375911 12122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f27d8833fb4991acd38c9713ccca64 in term 3.
I20260812 00:12:12.376091 12384 leader_election.cc:302] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9f27d8833fb4991acd38c9713ccca64, e8fe10ed12464d32b4234a170523cfaa; no voters: a6170ba2510b40ff8145edab8b083fcb
I20260812 00:12:12.376183 13326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 00:12:12.376298 13326 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [term 3 LEADER]: Becoming Leader. State: Replica: a9f27d8833fb4991acd38c9713ccca64, State: Running, Role: LEADER
I20260812 00:12:12.376401 13326 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } }
I20260812 00:12:12.376783 13331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f27d8833fb4991acd38c9713ccca64. Latest consensus state: current_term: 3 leader_uuid: "a9f27d8833fb4991acd38c9713ccca64" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:12.376888 13331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:12.377087 13334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:12.377615 13334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:12.377696 13334 catalog_manager.cc:1259] Loaded cluster ID: fb813337782d49bdab45c0a3717ea4e9
I20260812 00:12:12.377718 13334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:12.377861 13334 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:12.377911 13334 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Loaded TSK: 0
I20260812 00:12:12.378059 13334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:12.407476 12407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:12.409112 12407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:12.410365 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.410173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52532)
0812 00:12:12.410257 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.410300 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 00:12:12.410372 12122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [term 3 FOLLOWER]: Refusing update from remote peer a9f27d8833fb4991acd38c9713ccca64: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 00:12:12.410652 13331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 00:12:12.410807 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.410695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.0.0.1:52546)
0812 00:12:12.410747 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.410783 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 00:12:12.410916 12385 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 -> Peer a6170ba2510b40ff8145edab8b083fcb (127.31.227.62:46125): Couldn't send request to peer a6170ba2510b40ff8145edab8b083fcb. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 00:12:12.411331 13339 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f27d8833fb4991acd38c9713ccca64. Latest consensus state: current_term: 3 leader_uuid: "a9f27d8833fb4991acd38c9713ccca64" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:12.411454 13339 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:12.412066 13331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a9f27d8833fb4991acd38c9713ccca64" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:12.412158 13331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:12.412305 13341 mvcc.cc:205] Tried to move back new op lower bound from 7317477508750204928 to 7317477508614533120. Current Snapshot: MvccSnapshot[applied={T|T < 7317477508750204928}]
I20260812 00:12:12.412385 13339 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a9f27d8833fb4991acd38c9713ccca64" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a6170ba2510b40ff8145edab8b083fcb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46125 } } peers { permanent_uuid: "e8fe10ed12464d32b4234a170523cfaa" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 44461 } attrs { promote: false } } peers { permanent_uuid: "a9f27d8833fb4991acd38c9713ccca64" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 40935 } attrs { promote: false } } }
I20260812 00:12:12.412462 13339 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8fe10ed12464d32b4234a170523cfaa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:12.417454 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.414198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:37507 (local address 127.0.0.1:53778)
0812 00:12:12.414256 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.414284 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:12.414339 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:12.414647 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:12.414648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:12.414649 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:12.414712 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.414714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.415629 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.415630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.416501 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.416503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.417206 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.417207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.417277 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.417281 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:12.417337 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:12:12.417381 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":4}
I20260812 00:12:12.417482 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.414273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.31.227.2:37507)
0812 00:12:12.414472 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:12.414474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:12.414485 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:12.414515 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:12.414516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:12.414528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:12.414595 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:12.414900 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.415558 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.416590 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.417117 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.417204 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.417218 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:12.417361 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 00:12:12.417407 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260812 00:12:12.418072 12849 tablet_service.cc:1467] Processing CreateTablet for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 (DEFAULT_TABLE table=test-table [id=f157b415a41e4511a4c815fc4670a59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:12.418247 12849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:12.418284 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.414520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34555 (local address 127.0.0.1:59116)
0812 00:12:12.414854 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.414864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:12.414907 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:12.415138 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:12.415139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:12.415141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:12.415182 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.415183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.416037 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.416038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.417145 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.417148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.418032 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.418033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.418094 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.418098 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:12.418159 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:12:12.418196 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":69,"threads_started":1}
I20260812 00:12:12.418555 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.414424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:34005 (local address 127.0.0.1:35656)
0812 00:12:12.414653 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.414662 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:12.414717 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:12.415633 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:12.415634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:12.415636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:12.415669 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.415670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.416501 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.416502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.417584 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.417587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.418333 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.418334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.418402 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.418407 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:12.418458 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:12:12.418496 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260812 00:12:12.418588 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.414411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.31.227.1:34005)
0812 00:12:12.414518 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:12.414520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:12.414728 (+ 208us) server_negotiation.cc:407] Connection header received
0812 00:12:12.414767 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:12.414768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:12.414779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:12.414839 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:12.415746 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.416397 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.417688 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.418227 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.418286 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.418299 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:12.418465 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 00:12:12.418515 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260812 00:12:12.418848 12626 tablet_service.cc:1467] Processing CreateTablet for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 (DEFAULT_TABLE table=test-table [id=f157b415a41e4511a4c815fc4670a59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:12.418985 12626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:12.419847 13357 tablet_bootstrap.cc:492] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Bootstrap starting.
I20260812 00:12:12.420284 13357 tablet_bootstrap.cc:654] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:12.418570 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.414639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.31.227.3:34555)
0812 00:12:12.415003 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:12.415006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:12.415014 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:12.415039 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:12.415039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:12.415049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:12.415109 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:12.415250 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.415943 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.417255 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.417880 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.417952 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.417965 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:12.418242 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 00:12:12.418498 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":63,"threads_started":1}
I20260812 00:12:12.420421 13358 tablet_bootstrap.cc:492] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575: Bootstrap starting.
I20260812 00:12:12.420605 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 (DEFAULT_TABLE table=test-table [id=f157b415a41e4511a4c815fc4670a59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:12.420723 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:12.420812 13358 tablet_bootstrap.cc:654] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:12.421470 13358 tablet_bootstrap.cc:492] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575: No bootstrap required, opened a new log
I20260812 00:12:12.421557 13358 ts_tablet_manager.cc:1397] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:12.421684 13358 raft_consensus.cc:348] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.421757 13358 raft_consensus.cc:374] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:12.421777 13358 raft_consensus.cc:729] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5f406446794158bbd519f829730575, State: Initialized, Role: FOLLOWER
I20260812 00:12:12.421823 13358 consensus_queue.cc:260] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.422012 13357 tablet_bootstrap.cc:492] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: No bootstrap required, opened a new log
I20260812 00:12:12.422060 13357 ts_tablet_manager.cc:1397] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:12.422147 13358 ts_tablet_manager.cc:1428] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:12.422188 13357 raft_consensus.cc:348] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.422260 13357 raft_consensus.cc:374] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:12.422297 13357 raft_consensus.cc:729] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1d832ba2de433b9c890e5d935a4ca5, State: Initialized, Role: FOLLOWER
I20260812 00:12:12.422343 13357 consensus_queue.cc:260] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.422492 13357 ts_tablet_manager.cc:1428] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:12.422814 13365 tablet_bootstrap.cc:492] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Bootstrap starting.
I20260812 00:12:12.422880 12729 heartbeater.cc:503] Master 127.31.227.60:40935 was elected leader, sending a full tablet report...
I20260812 00:12:12.423252 13365 tablet_bootstrap.cc:654] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:12.423671 12966 heartbeater.cc:503] Master 127.31.227.60:40935 was elected leader, sending a full tablet report...
I20260812 00:12:12.423856 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.422394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:34195)
0812 00:12:12.423776 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.423815 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":71,"threads_started":1}
I20260812 00:12:12.424153 13365 tablet_bootstrap.cc:492] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: No bootstrap required, opened a new log
I20260812 00:12:12.424213 13365 ts_tablet_manager.cc:1397] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:12.424271 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.424053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:40545)
0812 00:12:12.424184 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.424227 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260812 00:12:12.424333 13365 raft_consensus.cc:348] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.424417 13365 raft_consensus.cc:374] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:12.424432 13365 raft_consensus.cc:729] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f22567680d4bf8b570e0ca4fd1be1f, State: Initialized, Role: FOLLOWER
I20260812 00:12:12.424470 13365 consensus_queue.cc:260] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.424595 13365 ts_tablet_manager.cc:1428] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:12.424788 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.422754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:58315)
0812 00:12:12.423607 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.423654 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":109,"threads_started":1}
I20260812 00:12:12.424975 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.424754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:51817)
0812 00:12:12.424886 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.424921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":92,"threads_started":1}
I20260812 00:12:12.425184 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.425005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:33949)
0812 00:12:12.425125 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.425154 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 00:12:12.425235 13188 heartbeater.cc:503] Master 127.31.227.60:40935 was elected leader, sending a full tablet report...
I20260812 00:12:12.425277 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.425191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:47413)
0812 00:12:12.425224 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.425250 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260812 00:12:12.475766 13192 tablet.cc:2369] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:12.581280 13363 raft_consensus.cc:482] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:12.581418 13363 raft_consensus.cc:504] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.581920 13363 leader_election.cc:288] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507), 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:12.584898 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.582228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:51333 (local address 127.31.227.3:34555)
0812 00:12:12.582277 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:12.582279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:12.582315 (+ 36us) server_negotiation.cc:407] Connection header received
0812 00:12:12.582391 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:12.582392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:12.582401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:12.582471 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:12.582644 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.583137 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.584164 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.584530 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.584617 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.584628 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:12.584786 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 00:12:12.584825 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:12.584925 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.582202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:43711 (local address 127.31.227.2:37507)
0812 00:12:12.582251 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:12.582253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:12.582299 (+ 46us) server_negotiation.cc:407] Connection header received
0812 00:12:12.582433 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:12.582434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:12.582451 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:12.582517 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:12.582675 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.583154 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.584178 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.584547 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.584630 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.584641 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:12.584802 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:12:12.584840 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:12.585011 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.582138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:37507 (local address 127.31.227.1:43711)
0812 00:12:12.582223 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.582237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:12.582321 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:12.582523 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:12.582524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:12.582526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:12.582602 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.582604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.583242 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.583244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.584073 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.584075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.584651 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.584653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.584723 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.584726 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:12.584795 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:12:12.584835 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":52}
I20260812 00:12:12.585258 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.582153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34555 (local address 127.31.227.1:51333)
0812 00:12:12.582232 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.582244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:12.582319 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:12.582475 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:12.582476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:12.582479 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:12.582567 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.582569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.583229 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.583230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.584065 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:12.584068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.584616 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.584617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:12.584707 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.584711 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:12.584777 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:12:12.584817 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":18}
I20260812 00:12:12.585291 12884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d852c328a4bfc9e3bc1ef0fc8f7c3" candidate_uuid: "9f5f406446794158bbd519f829730575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" is_pre_election: true
I20260812 00:12:12.585491 12884 raft_consensus.cc:2454] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5f406446794158bbd519f829730575 in term 0.
I20260812 00:12:12.585578 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d852c328a4bfc9e3bc1ef0fc8f7c3" candidate_uuid: "9f5f406446794158bbd519f829730575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" is_pre_election: true
I20260812 00:12:12.585677 13114 raft_consensus.cc:2454] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5f406446794158bbd519f829730575 in term 0.
I20260812 00:12:12.585716 12545 leader_election.cc:302] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f5f406446794158bbd519f829730575, ff1d832ba2de433b9c890e5d935a4ca5; no voters:
I20260812 00:12:12.585861 13363 raft_consensus.cc:2781] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:12.585906 13363 raft_consensus.cc:482] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:12.585937 13363 raft_consensus.cc:3037] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:12.586740 13363 raft_consensus.cc:504] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.586898 13363 leader_election.cc:288] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [CANDIDATE]: Term 1 election: Requested vote from peers ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507), 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555)
I20260812 00:12:12.587062 12884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d852c328a4bfc9e3bc1ef0fc8f7c3" candidate_uuid: "9f5f406446794158bbd519f829730575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1d832ba2de433b9c890e5d935a4ca5"
I20260812 00:12:12.587061 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d852c328a4bfc9e3bc1ef0fc8f7c3" candidate_uuid: "9f5f406446794158bbd519f829730575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f22567680d4bf8b570e0ca4fd1be1f"
I20260812 00:12:12.587131 13114 raft_consensus.cc:3037] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:12.587132 12884 raft_consensus.cc:3037] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:12.587845 13114 raft_consensus.cc:2454] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5f406446794158bbd519f829730575 in term 1.
I20260812 00:12:12.588006 12884 raft_consensus.cc:2454] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5f406446794158bbd519f829730575 in term 1.
I20260812 00:12:12.588033 12544 leader_election.cc:302] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f22567680d4bf8b570e0ca4fd1be1f, 9f5f406446794158bbd519f829730575; no voters:
I20260812 00:12:12.588148 13363 raft_consensus.cc:2781] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:12.588349 13363 raft_consensus.cc:686] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 1 LEADER]: Becoming Leader. State: Replica: 9f5f406446794158bbd519f829730575, State: Running, Role: LEADER
I20260812 00:12:12.588557 13363 consensus_queue.cc:237] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } }
I20260812 00:12:12.588754 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.588634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:47991)
0812 00:12:12.588690 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.588725 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:12:12.589350 12407 catalog_manager.cc:5495] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f5f406446794158bbd519f829730575 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "9f5f406446794158bbd519f829730575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f5f406446794158bbd519f829730575" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 34005 } health_report { overall_health: HEALTHY } } }
I20260812 00:12:12.600754 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.598493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.31.227.1:34005)
0812 00:12:12.598547 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:12.598548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:12.598555 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:12.598591 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:12.598592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:12.598610 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:12.598725 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:12.598867 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.599389 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.600095 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:12.600285 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:12.600339 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:12.600472 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:12.600582 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:12.600638 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:12.600677 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:12:12.600696 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:12:12.601894 13114 raft_consensus.cc:1261] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Refusing update from remote peer 9f5f406446794158bbd519f829730575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:12.601997 12884 raft_consensus.cc:1261] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Refusing update from remote peer 9f5f406446794158bbd519f829730575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:12.602176 13377 consensus_queue.cc:1035] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f22567680d4bf8b570e0ca4fd1be1f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:12.602329 13363 consensus_queue.cc:1035] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1d832ba2de433b9c890e5d935a4ca5" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:12.603509 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.603103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:40749)
0812 00:12:12.603419 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.603461 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260812 00:12:12.603981 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.603853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:34999)
0812 00:12:12.603903 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.603934 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 00:12:12.604436 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.604330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:34659)
0812 00:12:12.604376 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.604400 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 00:12:12.604792 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.604694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.3:42111)
0812 00:12:12.604733 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.604760 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 00:12:12.605017 13382 mvcc.cc:205] Tried to move back new op lower bound from 7317477509535285248 to 7317477509483442176. Current Snapshot: MvccSnapshot[applied={T|T < 7317477509535285248}]
I20260812 00:12:12.605437 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.605326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.1:43721)
0812 00:12:12.605360 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.605408 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 00:12:12.606202 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.606099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46125 (local address 127.31.227.2:37359)
0812 00:12:12.606151 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.606178 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 00:12:12.608148 12407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:12.608232 12407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:12.609608 12407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9f27d8833fb4991acd38c9713ccca64: Sending DeleteTablet for 3 replicas of tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3
I20260812 00:12:12.609946 12626 tablet_service.cc:1514] Processing DeleteTablet for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:12 UTC) from {username='oryx', principal='oryx/127.31.227.60@KRBTEST.COM'} at 127.0.0.1:35656
I20260812 00:12:12.609982 12849 tablet_service.cc:1514] Processing DeleteTablet for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:12 UTC) from {username='oryx', principal='oryx/127.31.227.60@KRBTEST.COM'} at 127.0.0.1:53778
I20260812 00:12:12.610168 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:12 UTC) from {username='oryx', principal='oryx/127.31.227.60@KRBTEST.COM'} at 127.0.0.1:59116
I20260812 00:12:12.610376 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 12340
I20260812 00:12:12.610432 13397 tablet_replica.cc:330] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575: stopping tablet replica
I20260812 00:12:12.610498 13397 raft_consensus.cc:2229] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:12.610500 13399 tablet_replica.cc:330] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: stopping tablet replica
I20260812 00:12:12.610560 13399 raft_consensus.cc:2229] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:12.610592 13397 raft_consensus.cc:2258] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 9f5f406446794158bbd519f829730575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:12.610610 13399 raft_consensus.cc:2258] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:12.610687 13398 tablet_replica.cc:330] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: stopping tablet replica
I20260812 00:12:12.610735 13398 raft_consensus.cc:2229] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:12.610966 13398 raft_consensus.cc:2258] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:12.618319 13398 ts_tablet_manager.cc:1905] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:12.620184 13398 ts_tablet_manager.cc:1918] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:12.620280 13398 log.cc:1196] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/289d852c328a4bfc9e3bc1ef0fc8f7c3
I20260812 00:12:12.620539 13398 ts_tablet_manager.cc:1939] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P ff1d832ba2de433b9c890e5d935a4ca5: Deleting consensus metadata
I20260812 00:12:12.620944 12385 catalog_manager.cc:4841] TS ff1d832ba2de433b9c890e5d935a4ca5 (127.31.227.2:37507): tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 (table test-table [id=f157b415a41e4511a4c815fc4670a59c]) successfully deleted
I20260812 00:12:12.622419 13399 ts_tablet_manager.cc:1905] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:12.624125 13399 ts_tablet_manager.cc:1918] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:12.624178 13399 log.cc:1196] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/289d852c328a4bfc9e3bc1ef0fc8f7c3
I20260812 00:12:12.624383 13399 ts_tablet_manager.cc:1939] T 289d852c328a4bfc9e3bc1ef0fc8f7c3 P 57f22567680d4bf8b570e0ca4fd1be1f: Deleting consensus metadata
I20260812 00:12:12.624735 12384 catalog_manager.cc:4841] TS 57f22567680d4bf8b570e0ca4fd1be1f (127.31.227.3:34555): tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3 (table test-table [id=f157b415a41e4511a4c815fc4670a59c]) successfully deleted
I20260812 00:12:12.631299 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:12.631161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:34005 (local address 127.0.0.1:35682)
0812 00:12:12.631230 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:12.631261 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:12:12.631436 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 12743
W20260812 00:12:12.631511 12381 catalog_manager.cc:4584] TS 9f5f406446794158bbd519f829730575 (127.31.227.1:34005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 289d852c328a4bfc9e3bc1ef0fc8f7c3: Network error: Client connection negotiation failed: client connection to 127.31.227.1:34005: connect: Connection refused (error 111)
I20260812 00:12:12.648424 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 12974
I20260812 00:12:12.661697 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 11873
I20260812 00:12:12.675384 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 12190
2026-08-12T00:12:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7111 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:12 dist-test-slave-zr1t krb5kdc[13405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:12 dist-test-slave-zr1t krb5kdc[13405](info): set up 2 sockets
Aug 12 00:12:12 dist-test-slave-zr1t krb5kdc[13405](info): commencing operation
krb5kdc: starting...
W20260812 00:12:13.972158 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab.
Entry for principal oryx/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab.
I20260812 00:12:14.091850 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:43405
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab
--principal=oryx/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:14.175865 13425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:14.176122 13425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:14.176163 13425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:14.177429 13425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:14.177497 13425 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:12:14.177562 13425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:14.177591 13425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:14.177632 13425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:14.177663 13425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:14.179224 13425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:43405
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.62.keytab
--principal=oryx/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:14.179529 13425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:14.179773 13425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:14.185289 13434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:14.185295 13441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:14.185384 13425 server_base.cc:1034] running on GCE node
W20260812 00:12:14.185295 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:14.185767 13425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:14.186087 13425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:14.187234 13425 hybrid_clock.cc:648] HybridClock initialized: now 1786493534187213 us; error 39 us; skew 500 ppm
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:14.193367 13425 init.cc:377] Logged in from keytab as oryx/127.31.227.62@KRBTEST.COM (short username oryx)
I20260812 00:12:14.193922 13425 webserver.cc:460] Webserver started at http://127.31.227.62:39975/ using document root <none> and password file <none>
I20260812 00:12:14.194168 13425 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:14.194214 13425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:14.194365 13425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:14.195492 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "6d7c3be8d2094b4b835a8f34c8875e17"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "e2c1af4d7f04da7ef6adb71396ed596a"
server_key_iv: "7740634a3923ab33c475345a399e7c7e"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.195871 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "6d7c3be8d2094b4b835a8f34c8875e17"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "e2c1af4d7f04da7ef6adb71396ed596a"
server_key_iv: "7740634a3923ab33c475345a399e7c7e"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.197245 13425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:12:14.198072 13450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.198264 13425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:14.198365 13425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "6d7c3be8d2094b4b835a8f34c8875e17"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "e2c1af4d7f04da7ef6adb71396ed596a"
server_key_iv: "7740634a3923ab33c475345a399e7c7e"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.198443 13425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:14.211432 13425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:14.213644 13425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:14.213809 13425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:14.219046 13425 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:43405
I20260812 00:12:14.219100 13566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:43405 every 8 connection(s)
I20260812 00:12:14.219430 13425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:14.220082 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.222486 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Bootstrap starting.
I20260812 00:12:14.223137 13568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.223462 13568 log.cc:824] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:14.224150 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: No bootstrap required, opened a new log
I20260812 00:12:14.226394 13568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43405 } }
I20260812 00:12:14.226583 13568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.226632 13568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7c3be8d2094b4b835a8f34c8875e17, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.226768 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 13425
I20260812 00:12:14.226831 13568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43405 } }
I20260812 00:12:14.226910 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:14.226935 13568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:14.226980 13568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:14.227030 13568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.227138 32652 external_mini_cluster.cc:1428] Setting key c8eb8567552ef054dc879d39bcc77340
I20260812 00:12:14.227866 13568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43405 } }
I20260812 00:12:14.227980 13568 leader_election.cc:302] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d7c3be8d2094b4b835a8f34c8875e17; no voters:
I20260812 00:12:14.228133 13568 leader_election.cc:288] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:14.228202 13573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:14.228401 13568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:14.228430 13573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [term 1 LEADER]: Becoming Leader. State: Replica: 6d7c3be8d2094b4b835a8f34c8875e17, State: Running, Role: LEADER
I20260812 00:12:14.229105 13573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43405 } }
I20260812 00:12:14.229867 13575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d7c3be8d2094b4b835a8f34c8875e17. Latest consensus state: current_term: 1 leader_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43405 } } }
I20260812 00:12:14.229949 13575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:14.230175 13576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7c3be8d2094b4b835a8f34c8875e17" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43405 } } }
I20260812 00:12:14.230288 13576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17 [sys.catalog]: This master's current role is: LEADER
W20260812 00:12:14.230656 13596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:14.230741 13596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:14.230861 13597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:14.231066 13597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:14.232702 13597 catalog_manager.cc:1347] Generated new cluster ID: 230aca10ef814b8090f9a30f0cec24ec
I20260812 00:12:14.232766 13597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:14.236630 13597 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:14.237123 13597 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:14.239761 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.227667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.31.227.62:43405)
0812 00:12:14.227911 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.227921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.227959 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:12:14.228084 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.228094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.228157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.228300 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.229093 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.230608 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.231688 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.232292 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.232403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.235405 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.235410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.235410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.235424 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.237688 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.238123 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.238127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.238128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.238255 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.238472 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.238473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.238474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.238763 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.238926 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.239388 (+ 462us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.239556 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":105,"threads_started":1}
I20260812 00:12:14.244113 13597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Generated new TSK 0
I20260812 00:12:14.244298 13597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab.
Entry for principal oryx/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab.
I20260812 00:12:14.274482 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:43405
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:14.361704 13606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:14.361908 13606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:14.361941 13606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:14.363134 13606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:14.363178 13606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:14.363198 13606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:12:14.363216 13606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:14.364763 13606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:14.365041 13606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:14.365289 13606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:14.370708 13616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:14.370860 13606 server_base.cc:1034] running on GCE node
W20260812 00:12:14.370721 13618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:14.370723 13621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:14.371284 13606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:14.371562 13606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:14.372726 13606 hybrid_clock.cc:648] HybridClock initialized: now 1786493534372707 us; error 36 us; skew 500 ppm
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:14.375831 13606 init.cc:377] Logged in from keytab as oryx/127.31.227.1@KRBTEST.COM (short username oryx)
I20260812 00:12:14.376398 13606 webserver.cc:460] Webserver started at http://127.31.227.1:45641/ using document root <none> and password file <none>
I20260812 00:12:14.376664 13606 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:14.376736 13606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:14.376896 13606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:14.378105 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "72c480d2608e4363a7884151d6b54ff5"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "75fd6ae8fcd2f38a236080420d9e1714"
server_key_iv: "6f2b6230d9a1aaa104fab5b592172835"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.378505 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "72c480d2608e4363a7884151d6b54ff5"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "75fd6ae8fcd2f38a236080420d9e1714"
server_key_iv: "6f2b6230d9a1aaa104fab5b592172835"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.379927 13606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:14.380851 13630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.381075 13606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:14.381182 13606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "72c480d2608e4363a7884151d6b54ff5"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "75fd6ae8fcd2f38a236080420d9e1714"
server_key_iv: "6f2b6230d9a1aaa104fab5b592172835"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.381263 13606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:14.388191 13606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:14.390588 13606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:14.390702 13606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:14.391177 13606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:14.391248 13606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.391300 13606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:14.391324 13606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.398329 13606 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:44631
I20260812 00:12:14.398411 13830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:44631 every 8 connection(s)
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:14.398718 13606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:14.400708 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 13606
I20260812 00:12:14.400880 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:14.401155 32652 external_mini_cluster.cc:1428] Setting key 5fd740c2d6f8d9a0094aaa6827b43d3e
I20260812 00:12:14.402516 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.391734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:52449 (local address 127.31.227.62:43405)
0812 00:12:14.391777 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.391779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.392012 (+ 233us) server_negotiation.cc:407] Connection header received
0812 00:12:14.392552 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.392554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.392570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.392649 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.393582 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.394461 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.396078 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.396419 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.396495 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.399349 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.399350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.399351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.399358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.400730 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.401310 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.401311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.401312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.401391 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.401744 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.401746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.401746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.401931 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.402007 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.402330 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.402428 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:12:14.402716 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.391730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.31.227.1:52449)
0812 00:12:14.391893 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.391929 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.392443 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.392721 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.392744 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.393131 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:14.393433 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.393441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.394566 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.394568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.395985 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.395988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.396512 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.396513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.396651 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.397261 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:14.397284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:14.399246 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:14.400864 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.400874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.400876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.401217 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.401473 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.401474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.401474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.401662 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.401997 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:14.402004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:14.402157 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.402289 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.402469 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260812 00:12:14.403250 13832 heartbeater.cc:347] Connected to a master server at 127.31.227.62:43405
I20260812 00:12:14.403363 13832 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:14.403589 13832 heartbeater.cc:511] Master 127.31.227.62:43405 requested a full tablet report, sending...
I20260812 00:12:14.404134 13477 ts_manager.cc:194] Registered new tserver with Master: 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631)
I20260812 00:12:14.404873 13477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:52449
WARNING: no policy specified for oryx/127.31.227.2@KRBTEST.COM; defaulting to no policy
I20260812 00:12:14.412262 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.405313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.31.227.1:60893)
0812 00:12:14.405940 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.405953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.406023 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.406127 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.406127 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.406353 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:14.406423 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.406426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.407069 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.407070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.407980 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.407984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.408336 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.408337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.408417 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.408900 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:14.408906 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:14.409351 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:14.411268 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.411269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.411270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.411494 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.411703 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.411704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.411704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.411776 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.412009 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:14.412009 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:14.412052 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.412109 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.412191 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":89,"threads_started":1}
I20260812 00:12:14.412315 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.405362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:60893 (local address 127.31.227.62:43405)
0812 00:12:14.405401 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.405403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.406028 (+ 625us) server_negotiation.cc:407] Connection header received
0812 00:12:14.406070 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.406071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.406081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.406121 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.406501 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.406967 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.408070 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.408256 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.408343 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.409474 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.409475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.409476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.409482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.411184 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.411582 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.411583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.411583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.411652 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.411841 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.411842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.411842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.411945 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.411998 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.412160 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.412235 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab.
Entry for principal oryx/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab.
I20260812 00:12:14.430585 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:43405
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:14.519414 13846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:14.519694 13846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:14.519738 13846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:14.521000 13846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:14.521075 13846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:14.521109 13846 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:12:14.521152 13846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:14.522877 13846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:14.523176 13846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:14.523451 13846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:12:14.529031 13846 server_base.cc:1034] running on GCE node
W20260812 00:12:14.529021 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:14.528983 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:14.528996 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:14.529378 13846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:14.529731 13846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:14.530882 13846 hybrid_clock.cc:648] HybridClock initialized: now 1786493534530862 us; error 43 us; skew 500 ppm
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:14.533638 13846 init.cc:377] Logged in from keytab as oryx/127.31.227.2@KRBTEST.COM (short username oryx)
I20260812 00:12:14.534114 13846 webserver.cc:460] Webserver started at http://127.31.227.2:39965/ using document root <none> and password file <none>
I20260812 00:12:14.534354 13846 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:14.534403 13846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:14.534545 13846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:14.535635 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "da94332a6bf34892933db847be7c7119"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "1e2377e908cf782fc622bd2269aa4fa6"
server_key_iv: "1ac1b57a5f9191d1bdd3cf4c511ea3d4"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.536010 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "da94332a6bf34892933db847be7c7119"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "1e2377e908cf782fc622bd2269aa4fa6"
server_key_iv: "1ac1b57a5f9191d1bdd3cf4c511ea3d4"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.537591 13846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:14.538393 13869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.538582 13846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:14.538689 13846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "da94332a6bf34892933db847be7c7119"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "1e2377e908cf782fc622bd2269aa4fa6"
server_key_iv: "1ac1b57a5f9191d1bdd3cf4c511ea3d4"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.538775 13846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:14.564286 13846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:14.566834 13846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:14.567013 13846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:14.567533 13846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:14.567569 13846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.567596 13846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:14.567646 13846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.575037 13846 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:40253
I20260812 00:12:14.575117 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:40253 every 8 connection(s)
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:14.575449 13846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:14.577239 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 13846
I20260812 00:12:14.577340 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:14.577641 32652 external_mini_cluster.cc:1428] Setting key 34095dc322e55205ec0897084380658c
I20260812 00:12:14.580834 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.568119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:45719 (local address 127.31.227.62:43405)
0812 00:12:14.568176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.568178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.568456 (+ 278us) server_negotiation.cc:407] Connection header received
0812 00:12:14.569055 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.569056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.569071 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.569117 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.570071 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.571025 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.572687 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.573005 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.573097 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.575920 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.575922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.575923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.575931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.577494 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.578086 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.578087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.578088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.578141 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.580157 (+ 2016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.580158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.580158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.580271 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.580333 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.580701 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.580771 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:12:14.581123 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.568111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.31.227.2:45719)
0812 00:12:14.568319 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.568357 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.568933 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.569210 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.569215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.569668 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:14.569973 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.569981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.571155 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.571157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.572586 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.572590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.573099 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.573100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.573202 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.573821 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:14.573832 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:14.575797 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:14.577620 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.577628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.577630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.578010 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.578207 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.578208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.578208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.578347 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.580348 (+ 2001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:14.580355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:14.580551 (+ 196us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.580678 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.580860 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260812 00:12:14.581830 14065 heartbeater.cc:347] Connected to a master server at 127.31.227.62:43405
I20260812 00:12:14.581919 14065 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:14.582118 14065 heartbeater.cc:511] Master 127.31.227.62:43405 requested a full tablet report, sending...
I20260812 00:12:14.582515 13477 ts_manager.cc:194] Registered new tserver with Master: da94332a6bf34892933db847be7c7119 (127.31.227.2:40253)
I20260812 00:12:14.582954 13477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:45719
WARNING: no policy specified for oryx/127.31.227.3@KRBTEST.COM; defaulting to no policy
I20260812 00:12:14.589730 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.583209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.31.227.2:45141)
0812 00:12:14.583781 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.583792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.583855 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.584017 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.584018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.584205 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:14.584255 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.584257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.585269 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.585270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.585894 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.585896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.586357 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.586358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.586406 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.586846 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:14.586850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:14.587133 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:14.588712 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.588713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.588714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.588881 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.589107 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.589108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.589108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.589177 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.589427 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:14.589428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:14.589465 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.589586 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.589664 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":518,"threads_started":1}
I20260812 00:12:14.589731 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.583237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:45141 (local address 127.31.227.62:43405)
0812 00:12:14.583281 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.583283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.583866 (+ 583us) server_negotiation.cc:407] Connection header received
0812 00:12:14.583897 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.583898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.583909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.584002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.584336 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.585181 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.585976 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.586279 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.586360 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.587225 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.587227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.587227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.587236 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.588621 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.588967 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.588968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.588968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.589030 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.589238 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.589238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.589239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.589368 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.589421 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.589614 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.589674 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab.
Entry for principal oryx/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab.
I20260812 00:12:14.606359 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:43405
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:14.693787 14080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:14.693989 14080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:14.694013 14080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:14.695230 14080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:14.695272 14080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:14.695292 14080 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:12:14.695364 14080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:14.696858 14080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.227.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:14.697100 14080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:14.697347 14080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:14.702360 14087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:14.702360 14088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:14.702407 14090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:14.702728 14080 server_base.cc:1034] running on GCE node
I20260812 00:12:14.702903 14080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:14.703109 14080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:14.704273 14080 hybrid_clock.cc:648] HybridClock initialized: now 1786493534704262 us; error 37 us; skew 500 ppm
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:14.706892 14080 init.cc:377] Logged in from keytab as oryx/127.31.227.3@KRBTEST.COM (short username oryx)
I20260812 00:12:14.707360 14080 webserver.cc:460] Webserver started at http://127.31.227.3:38727/ using document root <none> and password file <none>
I20260812 00:12:14.707598 14080 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:14.707666 14080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:14.707823 14080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:14.708948 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "33884777d0efc0518d3ee57785e8fd59"
server_key_iv: "3abedf4b085998882fdb16c82a1aa588"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.709324 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "33884777d0efc0518d3ee57785e8fd59"
server_key_iv: "3abedf4b085998882fdb16c82a1aa588"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.710804 14080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:14.711771 14101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.712006 14080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:14.712118 14080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
format_stamp: "Formatted at 2026-08-12 00:12:14 on dist-test-slave-zr1t"
server_key: "33884777d0efc0518d3ee57785e8fd59"
server_key_iv: "3abedf4b085998882fdb16c82a1aa588"
server_key_version: "encryptionkey@0"
I20260812 00:12:14.712205 14080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:14.747921 14080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:14.750219 14080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:14.750397 14080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:14.750979 14080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:14.751046 14080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.751094 14080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:14.751118 14080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:14.758939 14080 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:34609
I20260812 00:12:14.759055 14276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:34609 every 8 connection(s)
I20260812 00:12:14.759362 14080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:14.761363 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.751590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:54697 (local address 127.31.227.62:43405)
0812 00:12:14.751652 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.751654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.751886 (+ 232us) server_negotiation.cc:407] Connection header received
0812 00:12:14.752531 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.752533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.752552 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.752628 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.753584 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.754429 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.755537 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.755751 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.755832 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.758559 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.758560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.758561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.758568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.760186 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.760588 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.760588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.760589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.760645 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.760863 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.760863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.760863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.760966 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.761014 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.761238 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.761299 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:12:14.761600 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.751589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.31.227.3:54697)
0812 00:12:14.751761 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.751800 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.752335 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.752735 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.752740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.753127 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:14.753452 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.753459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.754539 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.754540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.755439 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.755442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.755856 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.755858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.755991 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.756541 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:14.756550 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:14.758446 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:14.760266 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.760271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.760273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.760519 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.760708 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.760709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.760710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.760809 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.761019 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:14.761024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:14.761139 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.761217 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.761366 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260812 00:12:14.762092 14279 heartbeater.cc:347] Connected to a master server at 127.31.227.62:43405
I20260812 00:12:14.762181 14279 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:14.762389 14279 heartbeater.cc:511] Master 127.31.227.62:43405 requested a full tablet report, sending...
I20260812 00:12:14.762774 13477 ts_manager.cc:194] Registered new tserver with Master: 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:14.763118 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 14080
I20260812 00:12:14.763227 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:14.763274 13477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:54697
I20260812 00:12:14.763458 32652 external_mini_cluster.cc:1428] Setting key 19a26d5dfac5ea7ba714cf5dafc2d773
I20260812 00:12:14.764392 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:14.769586 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.763197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.31.227.3:38363)
0812 00:12:14.763475 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.763488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.763568 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.763801 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.763802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.764066 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:14.764136 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.764138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.764851 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.764853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.765372 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.765374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.765820 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.765822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.765905 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.766370 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:14.766376 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:14.766663 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:14.768535 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.768536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.768536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.768716 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.768940 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:14.768940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:14.768941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:14.768997 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:14.769274 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:14.769275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:14.769313 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.769367 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.769445 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260812 00:12:14.769694 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.763269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:38363 (local address 127.31.227.62:43405)
0812 00:12:14.763310 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.763312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.763540 (+ 228us) server_negotiation.cc:407] Connection header received
0812 00:12:14.763723 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.763725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.763736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.763793 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.764216 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.764747 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.765462 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.765735 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.765807 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.766773 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.766774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.766775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.766782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.768422 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.768800 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.768801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.768802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.768868 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.769061 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.769062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.769062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.769209 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.769279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.769430 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.769555 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:12:14.788100 13572 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:14.782886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.31.227.62:43405)
0812 00:12:14.782957 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.782960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.782967 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:14.782990 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.782991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.782999 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.783054 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.783700 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.784171 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.784951 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.785133 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.785187 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.788022 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48136: BlockingRecv error: recv got EOF from 127.0.0.1:48136 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 00:12:14 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:14.795441 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.788867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.31.227.62:43405)
0812 00:12:14.788932 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.788933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.789020 (+ 87us) server_negotiation.cc:407] Connection header received
0812 00:12:14.789104 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.789105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.789115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.789168 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:14.789654 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.790127 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.790789 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.790979 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.791056 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.793160 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:14.793161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:14.793161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:14.793167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:14.794589 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.794888 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.794889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.794889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.794941 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:14.795090 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:14.795090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:14.795090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:14.795177 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:14.795245 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.795327 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.795379 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:12:14.797179 13477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:14.798825 13477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:14.805416 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.801202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44631 (local address 127.0.0.1:40514)
0812 00:12:14.801383 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.801425 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.801547 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.802161 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.802165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.802167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.802240 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.802242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.803202 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.803203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.804290 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.804293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.805058 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.805059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.805202 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.805220 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.805295 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.805343 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260812 00:12:14.805469 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.801419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.31.227.1:44631)
0812 00:12:14.802008 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.802012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.802023 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:14.802060 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.802062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.802076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.802151 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.802345 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.803109 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.804378 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.804978 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.805059 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.805134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.805330 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.805368 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":167,"threads_started":1}
I20260812 00:12:14.806183 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.801457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.31.227.2:40253)
0812 00:12:14.801665 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.801670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.801835 (+ 165us) server_negotiation.cc:407] Connection header received
0812 00:12:14.801979 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.801982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.801998 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.802077 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.802352 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.803541 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.805012 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.805799 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.805869 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.805935 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.806056 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.806090 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260812 00:12:14.806217 13722 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad20c9b54734573888ef4d70c4d3a3a (DEFAULT_TABLE table=test-table [id=572ed30f4a804b398347945692787cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:14.806463 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.801203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:40253 (local address 127.0.0.1:49424)
0812 00:12:14.801747 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.801754 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.801837 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.802167 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.802168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.802171 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.802248 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.802250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.803626 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.803627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.804915 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.804917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.805889 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.805891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.805953 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.805957 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.806038 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.806393 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":74,"thread_start_us":76,"threads_started":1}
I20260812 00:12:14.806631 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.801204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34609 (local address 127.0.0.1:45936)
0812 00:12:14.801654 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.801663 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.801701 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.802247 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.802248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.802249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.802297 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.802299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.803628 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.803628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.804914 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.804917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.806421 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.806422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.806486 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.806490 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.806522 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.806566 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":45,"thread_start_us":103,"threads_started":1}
I20260812 00:12:14.806651 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad20c9b54734573888ef4d70c4d3a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.807001 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.801422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.31.227.3:34609)
0812 00:12:14.801754 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.801757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.801766 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:14.801822 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.801824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.801842 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.802394 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.802465 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.803520 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.805004 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.806168 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.806758 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.806858 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.806885 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.806924 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260812 00:12:14.807649 14181 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad20c9b54734573888ef4d70c4d3a3a (DEFAULT_TABLE table=test-table [id=572ed30f4a804b398347945692787cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:14.808010 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad20c9b54734573888ef4d70c4d3a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.810027 13948 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad20c9b54734573888ef4d70c4d3a3a (DEFAULT_TABLE table=test-table [id=572ed30f4a804b398347945692787cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:14.810380 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad20c9b54734573888ef4d70c4d3a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.810719 14331 tablet_bootstrap.cc:492] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Bootstrap starting.
I20260812 00:12:14.811476 14331 tablet_bootstrap.cc:654] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.811781 14334 tablet_bootstrap.cc:492] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Bootstrap starting.
I20260812 00:12:14.811841 14331 log.cc:824] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:14.812577 14334 tablet_bootstrap.cc:654] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.812723 14331 tablet_bootstrap.cc:492] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: No bootstrap required, opened a new log
I20260812 00:12:14.812814 14331 ts_tablet_manager.cc:1397] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:14.812922 14334 log.cc:824] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:14.813745 14334 tablet_bootstrap.cc:492] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: No bootstrap required, opened a new log
I20260812 00:12:14.813812 14334 ts_tablet_manager.cc:1397] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:14.814561 14338 tablet_bootstrap.cc:492] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Bootstrap starting.
I20260812 00:12:14.815299 14338 tablet_bootstrap.cc:654] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.815337 14331 raft_consensus.cc:348] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.815567 14331 raft_consensus.cc:374] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.815613 14331 raft_consensus.cc:729] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.815653 14338 log.cc:824] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:14.815734 14331 consensus_queue.cc:260] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.815850 14334 raft_consensus.cc:348] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.816025 14334 raft_consensus.cc:374] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.816037 14331 ts_tablet_manager.cc:1428] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:14.816066 13832 heartbeater.cc:503] Master 127.31.227.62:43405 was elected leader, sending a full tablet report...
I20260812 00:12:14.816069 14334 raft_consensus.cc:729] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5457bd7d4c4eef833adfe79ef147d8, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.816186 14334 consensus_queue.cc:260] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.816462 14334 ts_tablet_manager.cc:1428] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:14.816505 14279 heartbeater.cc:503] Master 127.31.227.62:43405 was elected leader, sending a full tablet report...
I20260812 00:12:14.817096 14338 tablet_bootstrap.cc:492] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: No bootstrap required, opened a new log
I20260812 00:12:14.817167 14338 ts_tablet_manager.cc:1397] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:12:14.818840 14338 raft_consensus.cc:348] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.818957 14338 raft_consensus.cc:374] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.818979 14338 raft_consensus.cc:729] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94332a6bf34892933db847be7c7119, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.819054 14338 consensus_queue.cc:260] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.819239 14338 ts_tablet_manager.cc:1428] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:14.819329 14065 heartbeater.cc:503] Master 127.31.227.62:43405 was elected leader, sending a full tablet report...
W20260812 00:12:14.826086 14068 tablet.cc:2369] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:14.899507 13833 tablet.cc:2369] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:14.905233 14340 raft_consensus.cc:482] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:14.905435 14340 raft_consensus.cc:504] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.905829 14340 leader_election.cc:288] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609), da94332a6bf34892933db847be7c7119 (127.31.227.2:40253)
I20260812 00:12:14.908941 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.906151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:55391 (local address 127.31.227.2:40253)
0812 00:12:14.906199 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.906201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.906324 (+ 123us) server_negotiation.cc:407] Connection header received
0812 00:12:14.906418 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.906419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.906427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.906490 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.906670 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.907133 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.908195 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.908557 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.908644 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.908654 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.908814 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.908855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:14.908950 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.906143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38113 (local address 127.31.227.3:34609)
0812 00:12:14.906198 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.906200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.906287 (+ 87us) server_negotiation.cc:407] Connection header received
0812 00:12:14.906380 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.906381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.906391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.906470 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.906645 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.907137 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.908183 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.908575 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.908662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.908671 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.908838 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.908876 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:12:14.908985 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.906051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34609 (local address 127.31.227.1:38113)
0812 00:12:14.906207 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.906218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.906290 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.906478 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.906480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.906483 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.906565 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.906567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.907224 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.907225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.908090 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.908093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.908664 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.908665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.908744 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.908748 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.908829 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.908873 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260812 00:12:14.909013 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.906088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:40253 (local address 127.31.227.1:55391)
0812 00:12:14.906243 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.906253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.906342 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.906494 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.906495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.906497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.906587 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.906589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.907220 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.907221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.908081 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.908084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.908644 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.908646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.908736 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.908740 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.908809 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.908860 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":35,"thread_start_us":78,"threads_started":1}
I20260812 00:12:14.909412 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad20c9b54734573888ef4d70c4d3a3a" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119" is_pre_election: true
I20260812 00:12:14.909555 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad20c9b54734573888ef4d70c4d3a3a" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" is_pre_election: true
I20260812 00:12:14.909632 13982 raft_consensus.cc:2454] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 0.
I20260812 00:12:14.909729 14206 raft_consensus.cc:2454] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 0.
I20260812 00:12:14.909897 13633 leader_election.cc:302] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c480d2608e4363a7884151d6b54ff5, da94332a6bf34892933db847be7c7119; no voters:
I20260812 00:12:14.910028 14340 raft_consensus.cc:2781] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:14.910085 14340 raft_consensus.cc:482] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:14.910115 14340 raft_consensus.cc:3037] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.910822 14340 raft_consensus.cc:504] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.910979 14340 leader_election.cc:288] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609), da94332a6bf34892933db847be7c7119 (127.31.227.2:40253)
I20260812 00:12:14.911144 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad20c9b54734573888ef4d70c4d3a3a" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
I20260812 00:12:14.911211 14206 raft_consensus.cc:3037] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.911891 14206 raft_consensus.cc:2454] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 1.
I20260812 00:12:14.912026 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad20c9b54734573888ef4d70c4d3a3a" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119"
I20260812 00:12:14.912119 13634 leader_election.cc:302] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, 72c480d2608e4363a7884151d6b54ff5; no voters:
I20260812 00:12:14.912119 13982 raft_consensus.cc:3037] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.912204 14340 raft_consensus.cc:2781] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:14.912431 14340 raft_consensus.cc:686] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Running, Role: LEADER
I20260812 00:12:14.912562 14340 consensus_queue.cc:237] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } }
I20260812 00:12:14.913018 13982 raft_consensus.cc:2454] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 1.
I20260812 00:12:14.913641 13477 catalog_manager.cc:5495] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "72c480d2608e4363a7884151d6b54ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20260812 00:12:14.921917 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.919675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.31.227.1:44631)
0812 00:12:14.919738 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.919740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.919749 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:14.919783 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.919788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.919805 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.919872 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:14.920004 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.920476 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.921177 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.921368 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.921424 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.921647 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:14.921742 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:14.921803 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.921847 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.921866 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:12:14.923776 13982 raft_consensus.cc:1261] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Refusing update from remote peer 72c480d2608e4363a7884151d6b54ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:14.923784 14206 raft_consensus.cc:1261] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Refusing update from remote peer 72c480d2608e4363a7884151d6b54ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:14.924145 14340 consensus_queue.cc:1035] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:14.924278 14352 consensus_queue.cc:1035] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:14.926993 14362 mvcc.cc:205] Tried to move back new op lower bound from 7317477519044993024 to 7317477519002542080. Current Snapshot: MvccSnapshot[applied={T|T < 7317477519044993024}]
I20260812 00:12:14.927963 14361 mvcc.cc:205] Tried to move back new op lower bound from 7317477519044993024 to 7317477519002542080. Current Snapshot: MvccSnapshot[applied={T|T < 7317477519044993024 or (T in {7317477519044993024})}]
I20260812 00:12:14.929447 13477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:14.929566 13477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:14.930382 13477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Sending DeleteTablet for 3 replicas of tablet 0ad20c9b54734573888ef4d70c4d3a3a
I20260812 00:12:14.930809 14181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad20c9b54734573888ef4d70c4d3a3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:14 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260812 00:12:14.930814 13722 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad20c9b54734573888ef4d70c4d3a3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:14 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:40514
I20260812 00:12:14.931054 14373 tablet_replica.cc:330] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: stopping tablet replica
I20260812 00:12:14.931041 13948 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad20c9b54734573888ef4d70c4d3a3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:14 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260812 00:12:14.931176 14373 raft_consensus.cc:2229] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:14.931279 14373 raft_consensus.cc:2258] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:14.931514 13477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:14.931664 13477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:14.931607 14374 tablet_replica.cc:330] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: stopping tablet replica
I20260812 00:12:14.931800 14373 ts_tablet_manager.cc:1905] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:14.931962 14374 raft_consensus.cc:2229] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:14.932137 14374 raft_consensus.cc:2258] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:14.932840 14372 tablet_replica.cc:330] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: stopping tablet replica
I20260812 00:12:14.932950 14372 raft_consensus.cc:2229] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:14.932969 13722 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbb39570fa1473d9f5faad4e3d6ad42 (DEFAULT_TABLE table=test-table [id=d3aab8e1561748e78a25aa4b8fe2bc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:14.933152 14181 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbb39570fa1473d9f5faad4e3d6ad42 (DEFAULT_TABLE table=test-table [id=d3aab8e1561748e78a25aa4b8fe2bc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:14.933200 13948 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbb39570fa1473d9f5faad4e3d6ad42 (DEFAULT_TABLE table=test-table [id=d3aab8e1561748e78a25aa4b8fe2bc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:14.933288 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbb39570fa1473d9f5faad4e3d6ad42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.933329 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbb39570fa1473d9f5faad4e3d6ad42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.934137 14374 ts_tablet_manager.cc:1905] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:14.934824 14334 tablet_bootstrap.cc:492] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Bootstrap starting.
I20260812 00:12:14.934852 14338 tablet_bootstrap.cc:492] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Bootstrap starting.
I20260812 00:12:14.935302 14338 tablet_bootstrap.cc:654] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.935302 14334 tablet_bootstrap.cc:654] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.935819 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbb39570fa1473d9f5faad4e3d6ad42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:14.936094 14374 ts_tablet_manager.cc:1918] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:14.936211 14374 log.cc:1196] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/0ad20c9b54734573888ef4d70c4d3a3a
I20260812 00:12:14.936322 14372 raft_consensus.cc:2258] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:14.936638 14374 ts_tablet_manager.cc:1939] T 0ad20c9b54734573888ef4d70c4d3a3a P da94332a6bf34892933db847be7c7119: Deleting consensus metadata
I20260812 00:12:14.936821 14372 ts_tablet_manager.cc:1905] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:14.937183 13452 catalog_manager.cc:4841] TS da94332a6bf34892933db847be7c7119 (127.31.227.2:40253): tablet 0ad20c9b54734573888ef4d70c4d3a3a (table test-table [id=572ed30f4a804b398347945692787cab]) successfully deleted
I20260812 00:12:14.937338 14331 tablet_bootstrap.cc:492] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Bootstrap starting.
I20260812 00:12:14.937811 14331 tablet_bootstrap.cc:654] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:14.938452 14331 tablet_bootstrap.cc:492] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: No bootstrap required, opened a new log
I20260812 00:12:14.938512 14331 ts_tablet_manager.cc:1397] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:14.938669 14373 ts_tablet_manager.cc:1918] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:14.938691 14331 raft_consensus.cc:348] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.938793 14331 raft_consensus.cc:374] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.938783 14372 ts_tablet_manager.cc:1918] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:14.938822 14331 raft_consensus.cc:729] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.938850 14372 log.cc:1196] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/0ad20c9b54734573888ef4d70c4d3a3a
I20260812 00:12:14.938884 14331 consensus_queue.cc:260] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.938977 14331 ts_tablet_manager.cc:1428] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:14.939072 14373 log.cc:1196] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/0ad20c9b54734573888ef4d70c4d3a3a
I20260812 00:12:14.939123 14372 ts_tablet_manager.cc:1939] T 0ad20c9b54734573888ef4d70c4d3a3a P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting consensus metadata
I20260812 00:12:14.939471 14373 ts_tablet_manager.cc:1939] T 0ad20c9b54734573888ef4d70c4d3a3a P 72c480d2608e4363a7884151d6b54ff5: Deleting consensus metadata
I20260812 00:12:14.939533 13453 catalog_manager.cc:4841] TS 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609): tablet 0ad20c9b54734573888ef4d70c4d3a3a (table test-table [id=572ed30f4a804b398347945692787cab]) successfully deleted
I20260812 00:12:14.939935 13454 catalog_manager.cc:4841] TS 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631): tablet 0ad20c9b54734573888ef4d70c4d3a3a (table test-table [id=572ed30f4a804b398347945692787cab]) successfully deleted
I20260812 00:12:14.939976 14338 tablet_bootstrap.cc:492] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: No bootstrap required, opened a new log
I20260812 00:12:14.940030 14338 ts_tablet_manager.cc:1397] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 00:12:14.940181 14334 tablet_bootstrap.cc:492] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: No bootstrap required, opened a new log
I20260812 00:12:14.940191 14338 raft_consensus.cc:348] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.940260 14338 raft_consensus.cc:374] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.940253 14334 ts_tablet_manager.cc:1397] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 00:12:14.940289 14338 raft_consensus.cc:729] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94332a6bf34892933db847be7c7119, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.940330 14338 consensus_queue.cc:260] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.940423 14338 ts_tablet_manager.cc:1428] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:14.940441 14334 raft_consensus.cc:348] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.940519 14334 raft_consensus.cc:374] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:14.940546 14334 raft_consensus.cc:729] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5457bd7d4c4eef833adfe79ef147d8, State: Initialized, Role: FOLLOWER
I20260812 00:12:14.940600 14334 consensus_queue.cc:260] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.940686 14334 ts_tablet_manager.cc:1428] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:14.976444 14346 raft_consensus.cc:482] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:14.976554 14346 raft_consensus.cc:504] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.976984 14346 leader_election.cc:288] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631), 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:14.980134 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.977245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:32845 (local address 127.31.227.3:34609)
0812 00:12:14.977295 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.977298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.977309 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:14.977350 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.977351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.977362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.977422 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.977624 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.978250 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.979388 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.979759 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.979850 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.979860 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.980023 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.980064 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:14.980166 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.977179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34609 (local address 127.31.227.2:32845)
0812 00:12:14.977239 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.977251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.977295 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.977447 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.977448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.977449 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.977558 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.977561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.978341 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.978342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.979293 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.979296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.979847 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.979848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.979935 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.979938 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.980004 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.980051 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:12:14.980499 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbb39570fa1473d9f5faad4e3d6ad42" candidate_uuid: "da94332a6bf34892933db847be7c7119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" is_pre_election: true
I20260812 00:12:14.980607 14206 raft_consensus.cc:2454] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da94332a6bf34892933db847be7c7119 in term 0.
I20260812 00:12:14.980785 13873 leader_election.cc:302] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, da94332a6bf34892933db847be7c7119; no voters:
I20260812 00:12:14.980937 14346 raft_consensus.cc:2781] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:14.980988 14346 raft_consensus.cc:482] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:14.981019 14346 raft_consensus.cc:3037] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.981151 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.977421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:55067 (local address 127.31.227.1:44631)
0812 00:12:14.977467 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:14.977469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:14.977568 (+ 99us) server_negotiation.cc:407] Connection header received
0812 00:12:14.977707 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:14.977709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:14.977722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:14.977778 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:14.977981 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.978738 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.980155 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.980713 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.980758 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.980765 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:14.981016 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 00:12:14.981069 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:12:14.981362 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:14.977210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44631 (local address 127.31.227.2:55067)
0812 00:12:14.977497 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:14.977507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:14.977584 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:14.977796 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:14.977797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:14.977799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:14.977894 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.977906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.978861 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.978862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.980048 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:14.980051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:14.980829 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:14.980830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:14.980912 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:14.980917 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:14.981232 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 00:12:14.981276 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":71,"threads_started":1}
I20260812 00:12:14.981879 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbb39570fa1473d9f5faad4e3d6ad42" candidate_uuid: "da94332a6bf34892933db847be7c7119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c480d2608e4363a7884151d6b54ff5" is_pre_election: true
I20260812 00:12:14.982051 13753 raft_consensus.cc:2454] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da94332a6bf34892933db847be7c7119 in term 0.
I20260812 00:12:14.982156 14346 raft_consensus.cc:504] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.982347 14346 leader_election.cc:288] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [CANDIDATE]: Term 1 election: Requested vote from peers 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631), 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:14.982475 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbb39570fa1473d9f5faad4e3d6ad42" candidate_uuid: "da94332a6bf34892933db847be7c7119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c480d2608e4363a7884151d6b54ff5"
I20260812 00:12:14.982512 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbb39570fa1473d9f5faad4e3d6ad42" candidate_uuid: "da94332a6bf34892933db847be7c7119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
I20260812 00:12:14.982568 13753 raft_consensus.cc:3037] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.982581 14206 raft_consensus.cc:3037] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:14.983407 13753 raft_consensus.cc:2454] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da94332a6bf34892933db847be7c7119 in term 1.
I20260812 00:12:14.983399 14206 raft_consensus.cc:2454] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da94332a6bf34892933db847be7c7119 in term 1.
I20260812 00:12:14.983585 13873 leader_election.cc:302] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, da94332a6bf34892933db847be7c7119; no voters:
I20260812 00:12:14.983664 14346 raft_consensus.cc:2781] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:14.983902 14346 raft_consensus.cc:686] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 1 LEADER]: Becoming Leader. State: Replica: da94332a6bf34892933db847be7c7119, State: Running, Role: LEADER
I20260812 00:12:14.983989 14346 consensus_queue.cc:237] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:14.984808 13479 catalog_manager.cc:5495] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 reported cstate change: term changed from 0 to 1, leader changed from <none> to da94332a6bf34892933db847be7c7119 (127.31.227.2). New cstate: current_term: 1 leader_uuid: "da94332a6bf34892933db847be7c7119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } health_report { overall_health: UNKNOWN } } }
W20260812 00:12:15.010636 14280 tablet.cc:2369] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493534, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.62@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:15.018642 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.010324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:43405 (local address 127.0.0.1:48154)
0812 00:12:15.010554 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.010568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.010651 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.010845 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.010846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.011293 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:15.011354 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.011357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.012030 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.012031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.012871 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.012874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.013233 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.013235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.013311 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.014050 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:15.014056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:15.015762 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:15.017433 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:15.017441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:15.017443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:15.017706 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:15.017923 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:15.017924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:15.017925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:15.017984 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:15.018246 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:15.018254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:15.018298 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.018382 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.018505 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":133,"threads_started":1}
I20260812 00:12:15.018663 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.010412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.31.227.62:43405)
0812 00:12:15.010499 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.010501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.010656 (+ 155us) server_negotiation.cc:407] Connection header received
0812 00:12:15.010711 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.010712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.010725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.010787 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:15.011451 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.011933 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.012966 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.013145 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.013199 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.015866 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:15.015867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:15.015868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:15.015874 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:15.017322 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:15.017794 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:15.017795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:15.017796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:15.017854 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:15.018056 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:15.018057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:15.018057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:15.018169 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:15.018241 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.018415 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.018572 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":51}
I20260812 00:12:15.020959 13479 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:48154:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:12:15.021160 13479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:15.022782 13722 tablet_service.cc:1467] Processing CreateTablet for tablet 08e65fd7560b484389f35ba7e11c9a1c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=889ed1e131e04771925f32851ef43854]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:12:15.022826 13948 tablet_service.cc:1467] Processing CreateTablet for tablet 08e65fd7560b484389f35ba7e11c9a1c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=889ed1e131e04771925f32851ef43854]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:12:15.022924 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e65fd7560b484389f35ba7e11c9a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.023000 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e65fd7560b484389f35ba7e11c9a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.023986 14331 tablet_bootstrap.cc:492] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: Bootstrap starting.
I20260812 00:12:15.024405 14331 tablet_bootstrap.cc:654] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.024546 14338 tablet_bootstrap.cc:492] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: Bootstrap starting.
I20260812 00:12:15.024744 14181 tablet_service.cc:1467] Processing CreateTablet for tablet 08e65fd7560b484389f35ba7e11c9a1c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=889ed1e131e04771925f32851ef43854]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:12:15.024902 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e65fd7560b484389f35ba7e11c9a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.024942 14338 tablet_bootstrap.cc:654] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.025593 14338 tablet_bootstrap.cc:492] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: No bootstrap required, opened a new log
I20260812 00:12:15.025636 14338 ts_tablet_manager.cc:1397] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:15.025767 14338 raft_consensus.cc:348] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.025816 14338 raft_consensus.cc:374] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.025832 14338 raft_consensus.cc:729] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94332a6bf34892933db847be7c7119, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.025895 14338 consensus_queue.cc:260] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.026018 14338 ts_tablet_manager.cc:1428] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:15.026053 14346 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.026115 14346 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.026471 14334 tablet_bootstrap.cc:492] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: Bootstrap starting.
I20260812 00:12:15.026979 14334 tablet_bootstrap.cc:654] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.027738 14334 tablet_bootstrap.cc:492] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: No bootstrap required, opened a new log
I20260812 00:12:15.027819 14334 ts_tablet_manager.cc:1397] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:15.027971 14334 raft_consensus.cc:348] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.028033 14334 raft_consensus.cc:374] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.028051 14334 raft_consensus.cc:729] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5457bd7d4c4eef833adfe79ef147d8, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.028100 14334 consensus_queue.cc:260] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.028187 14334 ts_tablet_manager.cc:1428] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:15.028261 14342 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.028415 14342 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.028947 14331 tablet_bootstrap.cc:492] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: No bootstrap required, opened a new log
I20260812 00:12:15.029022 14331 ts_tablet_manager.cc:1397] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 00:12:15.029213 14331 raft_consensus.cc:348] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.029285 14331 raft_consensus.cc:374] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.029300 14331 raft_consensus.cc:729] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.029353 14331 consensus_queue.cc:260] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.029450 14331 ts_tablet_manager.cc:1428] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:15.029491 14367 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.029587 14367 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.058426 14342 raft_consensus.cc:482] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:15.058516 14342 raft_consensus.cc:504] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.058869 14342 leader_election.cc:288] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631), da94332a6bf34892933db847be7c7119 (127.31.227.2:40253)
I20260812 00:12:15.061935 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.059018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:40253 (local address 127.31.227.3:56591)
0812 00:12:15.059077 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.059088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.059132 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.059333 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.059334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.059335 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:15.059434 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.059437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.060200 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.060201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.060996 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.060999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.061662 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.061663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.061746 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.061750 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.061814 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.061862 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:12:15.061935 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.059069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:56591 (local address 127.31.227.2:40253)
0812 00:12:15.059115 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.059117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.059127 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:15.059220 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.059221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.059236 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.059283 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:15.059525 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.060097 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.061101 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.061576 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.061641 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.061651 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.061822 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.061863 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:15.062288 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.059204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:60397 (local address 127.31.227.1:44631)
0812 00:12:15.059248 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.059250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.059261 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:15.059346 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.059347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.059356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.059387 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:15.059605 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.060072 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.061396 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.061814 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.061897 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.061905 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.062168 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.062212 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:12:15.062340 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.059021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44631 (local address 127.31.227.3:60397)
0812 00:12:15.059199 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.059209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.059255 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.059453 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.059454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.059456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:15.059526 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.059527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.060161 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.060163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.061295 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.061298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.061902 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.061903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.061977 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.061981 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.062135 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.062183 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260812 00:12:15.062340 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e65fd7560b484389f35ba7e11c9a1c" candidate_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119" is_pre_election: true
I20260812 00:12:15.062453 13982 raft_consensus.cc:2454] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5457bd7d4c4eef833adfe79ef147d8 in term 0.
I20260812 00:12:15.062568 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e65fd7560b484389f35ba7e11c9a1c" candidate_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c480d2608e4363a7884151d6b54ff5" is_pre_election: true
I20260812 00:12:15.062662 13753 raft_consensus.cc:2454] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5457bd7d4c4eef833adfe79ef147d8 in term 0.
I20260812 00:12:15.062647 14104 leader_election.cc:302] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, da94332a6bf34892933db847be7c7119; no voters:
I20260812 00:12:15.062779 14342 raft_consensus.cc:2781] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:15.062860 14342 raft_consensus.cc:482] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:15.062892 14342 raft_consensus.cc:3037] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.063596 14342 raft_consensus.cc:504] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.063758 14342 leader_election.cc:288] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [CANDIDATE]: Term 1 election: Requested vote from peers 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631), da94332a6bf34892933db847be7c7119 (127.31.227.2:40253)
I20260812 00:12:15.063963 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e65fd7560b484389f35ba7e11c9a1c" candidate_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119"
I20260812 00:12:15.063953 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e65fd7560b484389f35ba7e11c9a1c" candidate_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c480d2608e4363a7884151d6b54ff5"
I20260812 00:12:15.064035 13982 raft_consensus.cc:3037] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.064035 13753 raft_consensus.cc:3037] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.064795 13753 raft_consensus.cc:2454] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5457bd7d4c4eef833adfe79ef147d8 in term 1.
I20260812 00:12:15.064846 13982 raft_consensus.cc:2454] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5457bd7d4c4eef833adfe79ef147d8 in term 1.
I20260812 00:12:15.065007 14104 leader_election.cc:302] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, da94332a6bf34892933db847be7c7119; no voters:
I20260812 00:12:15.065142 14342 raft_consensus.cc:2781] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:15.065286 14342 raft_consensus.cc:686] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5457bd7d4c4eef833adfe79ef147d8, State: Running, Role: LEADER
I20260812 00:12:15.065369 14342 consensus_queue.cc:237] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.065771 14395 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: TxnStatusTablet state changed. Reason: New leader 2d5457bd7d4c4eef833adfe79ef147d8. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.065871 14395 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:12:15.066046 14398 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:12:15.066138 14398 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:12:15.066140 13479 catalog_manager.cc:5495] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3). New cstate: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } health_report { overall_health: HEALTHY } } }
I20260812 00:12:15.078703 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.075947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.31.227.3:34609)
0812 00:12:15.076005 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.076007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.076017 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:15.076094 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.076094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.076109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.076166 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.076303 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.076829 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.077976 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.078153 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.078203 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.078407 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.078496 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.078546 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.078625 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.078652 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:12:15.078781 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.075843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34609 (local address 127.0.0.1:45946)
0812 00:12:15.075936 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.075949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.076003 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.076190 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.076191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.076192 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:12:15.076243 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.076246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.076906 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.076907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.077882 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.077884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.078224 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.078225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.078293 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.078311 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.078557 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.078566 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.078623 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.078684 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 00:12:15.079728 13982 raft_consensus.cc:1261] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Refusing update from remote peer 2d5457bd7d4c4eef833adfe79ef147d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.079851 13753 raft_consensus.cc:1261] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Refusing update from remote peer 2d5457bd7d4c4eef833adfe79ef147d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.079993 14395 consensus_queue.cc:1035] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.080154 14342 consensus_queue.cc:1035] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.080597 14346 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: TxnStatusTablet state changed. Reason: New leader 2d5457bd7d4c4eef833adfe79ef147d8. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.080689 14346 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.081174 14342 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.081267 14342 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:12:15.081727 14367 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: TxnStatusTablet state changed. Reason: New leader 2d5457bd7d4c4eef833adfe79ef147d8. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.081818 14367 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.081840 14342 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.081921 14342 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 2d5457bd7d4c4eef833adfe79ef147d8: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:12:15.082201 14367 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.082274 14367 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P 72c480d2608e4363a7884151d6b54ff5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.082841 14346 tablet_replica.cc:439] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } } }
I20260812 00:12:15.082919 14346 tablet_replica.cc:442] T 08e65fd7560b484389f35ba7e11c9a1c P da94332a6bf34892933db847be7c7119: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:12:15.086400 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.083959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.31.227.2:40253)
0812 00:12:15.084002 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.084004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.084013 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:15.084085 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.084085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.084103 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.084166 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.084309 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.084840 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.085612 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.085817 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.085901 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.086071 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.086176 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.086252 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.086309 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.086327 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:15.091272 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.088811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34609 (local address 127.31.227.2:53177)
0812 00:12:15.088900 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.088911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.088948 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.089080 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.089081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.089082 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:12:15.089126 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.089127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.089920 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.089920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.090488 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.090490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.090939 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.090939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.090985 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.090999 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.091164 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.091166 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.091198 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.091228 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 00:12:15.091281 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.088877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:53177 (local address 127.31.227.3:34609)
0812 00:12:15.088931 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.088932 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.088941 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:15.089006 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.089007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.089015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.089079 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.089160 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.089849 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.090579 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.090864 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.090933 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.091074 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.091105 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.091158 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.091201 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.091219 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:12:15.093453 14206 raft_consensus.cc:1261] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Refusing update from remote peer da94332a6bf34892933db847be7c7119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.093613 13753 raft_consensus.cc:1261] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Refusing update from remote peer da94332a6bf34892933db847be7c7119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.093739 14380 consensus_queue.cc:1035] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.093848 14346 consensus_queue.cc:1035] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.097044 14359 mvcc.cc:205] Tried to move back new op lower bound from 7317477519741308928 to 7317477519295062016. Current Snapshot: MvccSnapshot[applied={T|T < 7317477519741308928}]
I20260812 00:12:15.102582 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.099842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58985 (local address 127.31.227.2:40253)
0812 00:12:15.099886 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.099888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.099941 (+ 53us) server_negotiation.cc:407] Connection header received
0812 00:12:15.100032 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.100032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.100042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.100094 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.100222 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.100672 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.101840 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.102126 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.102207 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.102387 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.102418 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.102459 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.102516 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.102532 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:12:15.102633 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.099826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:40253 (local address 127.31.227.3:58985)
0812 00:12:15.099901 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.099910 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.099956 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.100100 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.100101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.100103 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:12:15.100166 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.100168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.100739 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.100740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.101752 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.101755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.102205 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.102207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.102279 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.102309 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.102464 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.102466 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.102514 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.102559 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:12:15.109958 13477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:15.110074 13477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:15.110807 13477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Sending DeleteTablet for 3 replicas of tablet 3dbb39570fa1473d9f5faad4e3d6ad42
I20260812 00:12:15.111109 14181 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dbb39570fa1473d9f5faad4e3d6ad42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260812 00:12:15.111217 14372 tablet_replica.cc:330] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: stopping tablet replica
I20260812 00:12:15.111231 13722 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dbb39570fa1473d9f5faad4e3d6ad42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:40514
I20260812 00:12:15.111274 14372 raft_consensus.cc:2229] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:15.111311 14372 raft_consensus.cc:2258] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.111330 14373 tablet_replica.cc:330] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: stopping tablet replica
I20260812 00:12:15.111379 14373 raft_consensus.cc:2229] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:15.111419 14373 raft_consensus.cc:2258] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.111841 14373 ts_tablet_manager.cc:1905] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.113607 14373 ts_tablet_manager.cc:1918] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:12:15.113664 14373 log.cc:1196] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/3dbb39570fa1473d9f5faad4e3d6ad42
I20260812 00:12:15.113895 14373 ts_tablet_manager.cc:1939] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 72c480d2608e4363a7884151d6b54ff5: Deleting consensus metadata
I20260812 00:12:15.114351 13454 catalog_manager.cc:4841] TS 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631): tablet 3dbb39570fa1473d9f5faad4e3d6ad42 (table test-table [id=d3aab8e1561748e78a25aa4b8fe2bc77]) successfully deleted
I20260812 00:12:15.114516 13948 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dbb39570fa1473d9f5faad4e3d6ad42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260812 00:12:15.114598 14374 tablet_replica.cc:330] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: stopping tablet replica
I20260812 00:12:15.114676 14374 raft_consensus.cc:2229] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:15.114766 14374 raft_consensus.cc:2258] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.115130 14374 ts_tablet_manager.cc:1905] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.116688 14374 ts_tablet_manager.cc:1918] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:12:15.116753 14374 log.cc:1196] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/3dbb39570fa1473d9f5faad4e3d6ad42
I20260812 00:12:15.116981 14374 ts_tablet_manager.cc:1939] T 3dbb39570fa1473d9f5faad4e3d6ad42 P da94332a6bf34892933db847be7c7119: Deleting consensus metadata
I20260812 00:12:15.117352 13452 catalog_manager.cc:4841] TS da94332a6bf34892933db847be7c7119 (127.31.227.2:40253): tablet 3dbb39570fa1473d9f5faad4e3d6ad42 (table test-table [id=d3aab8e1561748e78a25aa4b8fe2bc77]) successfully deleted
I20260812 00:12:15.117482 14372 ts_tablet_manager.cc:1905] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.119074 14372 ts_tablet_manager.cc:1918] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:12:15.119143 14372 log.cc:1196] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/3dbb39570fa1473d9f5faad4e3d6ad42
I20260812 00:12:15.119351 14372 ts_tablet_manager.cc:1939] T 3dbb39570fa1473d9f5faad4e3d6ad42 P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting consensus metadata
I20260812 00:12:15.119796 13453 catalog_manager.cc:4841] TS 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609): tablet 3dbb39570fa1473d9f5faad4e3d6ad42 (table test-table [id=d3aab8e1561748e78a25aa4b8fe2bc77]) successfully deleted
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[13405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:12:15.148849 13572 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:15.141094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.31.227.62:43405)
0812 00:12:15.141149 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.141152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.141743 (+ 591us) server_negotiation.cc:407] Connection header received
0812 00:12:15.141810 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.141812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.141827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.141903 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:15.142793 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.143595 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.144734 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.145052 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.145127 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.148781 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48158: BlockingRecv error: recv got EOF from 127.0.0.1:48158 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[13405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM
I20260812 00:12:15.159487 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.150328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.31.227.62:43405)
0812 00:12:15.150512 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.150515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.150525 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:15.150642 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.150643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.150655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.150722 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:15.151323 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.152057 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.153094 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.153384 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.153465 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.156327 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:15.156328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:15.156329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:15.156338 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:15.158159 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:15.158572 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:15.158573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:15.158573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:15.158643 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:15.158872 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:15.158873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:15.158874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:15.159028 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:15.159131 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.159291 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.159385 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260812 00:12:15.161950 13477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:15.162142 13477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:15.163774 13948 tablet_service.cc:1467] Processing CreateTablet for tablet d063324fbb3e42d7bc23c789fc3f3141 (DEFAULT_TABLE table=test-table [id=ccb4226502fd43f99f63ae2286302fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:15.163853 14181 tablet_service.cc:1467] Processing CreateTablet for tablet d063324fbb3e42d7bc23c789fc3f3141 (DEFAULT_TABLE table=test-table [id=ccb4226502fd43f99f63ae2286302fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:15.163949 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d063324fbb3e42d7bc23c789fc3f3141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.163991 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d063324fbb3e42d7bc23c789fc3f3141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.165431 14334 tablet_bootstrap.cc:492] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Bootstrap starting.
I20260812 00:12:15.165445 14338 tablet_bootstrap.cc:492] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Bootstrap starting.
I20260812 00:12:15.165911 14338 tablet_bootstrap.cc:654] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.166144 14334 tablet_bootstrap.cc:654] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.166828 14334 tablet_bootstrap.cc:492] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: No bootstrap required, opened a new log
I20260812 00:12:15.166882 14334 ts_tablet_manager.cc:1397] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:15.167091 14334 raft_consensus.cc:348] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.167181 14334 raft_consensus.cc:374] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.167209 14334 raft_consensus.cc:729] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5457bd7d4c4eef833adfe79ef147d8, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.167255 14334 consensus_queue.cc:260] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.167354 14334 ts_tablet_manager.cc:1428] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:15.167603 14338 tablet_bootstrap.cc:492] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: No bootstrap required, opened a new log
I20260812 00:12:15.167654 14338 ts_tablet_manager.cc:1397] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:15.167814 14338 raft_consensus.cc:348] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.167876 14338 raft_consensus.cc:374] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.167896 14338 raft_consensus.cc:729] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94332a6bf34892933db847be7c7119, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.167917 13722 tablet_service.cc:1467] Processing CreateTablet for tablet d063324fbb3e42d7bc23c789fc3f3141 (DEFAULT_TABLE table=test-table [id=ccb4226502fd43f99f63ae2286302fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:15.167937 14338 consensus_queue.cc:260] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.168015 14338 ts_tablet_manager.cc:1428] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:15.168046 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d063324fbb3e42d7bc23c789fc3f3141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.169480 14331 tablet_bootstrap.cc:492] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Bootstrap starting.
I20260812 00:12:15.169885 14331 tablet_bootstrap.cc:654] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.170464 14331 tablet_bootstrap.cc:492] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: No bootstrap required, opened a new log
I20260812 00:12:15.170512 14331 ts_tablet_manager.cc:1397] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:15.170616 14331 raft_consensus.cc:348] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.170683 14331 raft_consensus.cc:374] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.170704 14331 raft_consensus.cc:729] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.170743 14331 consensus_queue.cc:260] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.170832 14331 ts_tablet_manager.cc:1428] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:15.194469 14367 raft_consensus.cc:482] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:15.194607 14367 raft_consensus.cc:504] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.194789 14367 leader_election.cc:288] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da94332a6bf34892933db847be7c7119 (127.31.227.2:40253), 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:15.195042 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d063324fbb3e42d7bc23c789fc3f3141" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119" is_pre_election: true
I20260812 00:12:15.195166 13982 raft_consensus.cc:2454] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 0.
I20260812 00:12:15.195115 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d063324fbb3e42d7bc23c789fc3f3141" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" is_pre_election: true
I20260812 00:12:15.195223 14206 raft_consensus.cc:2454] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 0.
I20260812 00:12:15.195436 13634 leader_election.cc:302] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, 72c480d2608e4363a7884151d6b54ff5; no voters:
I20260812 00:12:15.195609 14367 raft_consensus.cc:2781] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:15.195673 14367 raft_consensus.cc:482] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:15.195696 14367 raft_consensus.cc:3037] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.196740 14367 raft_consensus.cc:504] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.196892 14367 leader_election.cc:288] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 election: Requested vote from peers da94332a6bf34892933db847be7c7119 (127.31.227.2:40253), 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:15.197079 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d063324fbb3e42d7bc23c789fc3f3141" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119"
I20260812 00:12:15.197155 13982 raft_consensus.cc:3037] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.197150 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d063324fbb3e42d7bc23c789fc3f3141" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
I20260812 00:12:15.197227 14206 raft_consensus.cc:3037] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.198033 13982 raft_consensus.cc:2454] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 1.
I20260812 00:12:15.198071 14206 raft_consensus.cc:2454] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 1.
I20260812 00:12:15.198251 13634 leader_election.cc:302] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, 72c480d2608e4363a7884151d6b54ff5; no voters:
I20260812 00:12:15.198401 14367 raft_consensus.cc:2781] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:15.198454 14367 raft_consensus.cc:686] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Running, Role: LEADER
I20260812 00:12:15.198505 14367 consensus_queue.cc:237] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.199120 13479 catalog_manager.cc:5495] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "72c480d2608e4363a7884151d6b54ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } health_report { overall_health: UNKNOWN } } }
I20260812 00:12:15.210071 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.206730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.31.227.1:44631)
0812 00:12:15.206851 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.206853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.206860 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:15.206887 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.206888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.206899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.206945 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.207131 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.207896 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.209021 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.209322 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.209393 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.209634 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.209672 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.209717 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.209963 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.209987 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 00:12:15.210973 13982 raft_consensus.cc:1261] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Refusing update from remote peer 72c480d2608e4363a7884151d6b54ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.211366 14206 raft_consensus.cc:1261] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Refusing update from remote peer 72c480d2608e4363a7884151d6b54ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.211310 14367 consensus_queue.cc:1035] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.211791 14367 consensus_queue.cc:1035] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.215277 13477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:15.215420 13477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:15.216058 13477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Sending DeleteTablet for 3 replicas of tablet d063324fbb3e42d7bc23c789fc3f3141
I20260812 00:12:15.216472 14181 tablet_service.cc:1514] Processing DeleteTablet for tablet d063324fbb3e42d7bc23c789fc3f3141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260812 00:12:15.216567 14372 tablet_replica.cc:330] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: stopping tablet replica
I20260812 00:12:15.216615 14372 raft_consensus.cc:2229] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:15.216621 13948 tablet_service.cc:1514] Processing DeleteTablet for tablet d063324fbb3e42d7bc23c789fc3f3141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260812 00:12:15.216657 14372 raft_consensus.cc:2258] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.216712 14374 tablet_replica.cc:330] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: stopping tablet replica
I20260812 00:12:15.216761 14374 raft_consensus.cc:2229] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:15.216806 14374 raft_consensus.cc:2258] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.217041 13722 tablet_service.cc:1514] Processing DeleteTablet for tablet d063324fbb3e42d7bc23c789fc3f3141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:40514
I20260812 00:12:15.217166 14373 tablet_replica.cc:330] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: stopping tablet replica
I20260812 00:12:15.217233 14373 raft_consensus.cc:2229] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:15.217301 14373 raft_consensus.cc:2258] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.217453 14374 ts_tablet_manager.cc:1905] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.218009 13479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:15.218170 13479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:15.219190 14374 ts_tablet_manager.cc:1918] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:15.219254 14374 log.cc:1196] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/d063324fbb3e42d7bc23c789fc3f3141
I20260812 00:12:15.219492 14374 ts_tablet_manager.cc:1939] T d063324fbb3e42d7bc23c789fc3f3141 P da94332a6bf34892933db847be7c7119: Deleting consensus metadata
I20260812 00:12:15.223794 14373 ts_tablet_manager.cc:1905] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.224157 14181 tablet_service.cc:1467] Processing CreateTablet for tablet fe13cf30e4f74c429d0428421981430d (DEFAULT_TABLE table=test-table [id=9caa149ff8c04fa78e037a1421f46798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:15.224313 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe13cf30e4f74c429d0428421981430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.225291 14373 ts_tablet_manager.cc:1918] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:15.225346 14373 log.cc:1196] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/d063324fbb3e42d7bc23c789fc3f3141
I20260812 00:12:15.225785 14334 tablet_bootstrap.cc:492] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Bootstrap starting.
I20260812 00:12:15.226195 14334 tablet_bootstrap.cc:654] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.226253 13722 tablet_service.cc:1467] Processing CreateTablet for tablet fe13cf30e4f74c429d0428421981430d (DEFAULT_TABLE table=test-table [id=9caa149ff8c04fa78e037a1421f46798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:15.226419 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe13cf30e4f74c429d0428421981430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.227135 13948 tablet_service.cc:1467] Processing CreateTablet for tablet fe13cf30e4f74c429d0428421981430d (DEFAULT_TABLE table=test-table [id=9caa149ff8c04fa78e037a1421f46798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:15.227255 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe13cf30e4f74c429d0428421981430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:15.227634 14331 tablet_bootstrap.cc:492] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: Bootstrap starting.
I20260812 00:12:15.228506 14338 tablet_bootstrap.cc:492] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Bootstrap starting.
I20260812 00:12:15.228514 13452 catalog_manager.cc:4841] TS da94332a6bf34892933db847be7c7119 (127.31.227.2:40253): tablet d063324fbb3e42d7bc23c789fc3f3141 (table test-table [id=ccb4226502fd43f99f63ae2286302fe3]) successfully deleted
I20260812 00:12:15.228916 14373 ts_tablet_manager.cc:1939] T d063324fbb3e42d7bc23c789fc3f3141 P 72c480d2608e4363a7884151d6b54ff5: Deleting consensus metadata
I20260812 00:12:15.228974 14338 tablet_bootstrap.cc:654] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.229552 14331 tablet_bootstrap.cc:654] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:15.229627 14372 ts_tablet_manager.cc:1905] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.230178 13454 catalog_manager.cc:4841] TS 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631): tablet d063324fbb3e42d7bc23c789fc3f3141 (table test-table [id=ccb4226502fd43f99f63ae2286302fe3]) successfully deleted
I20260812 00:12:15.230669 14338 tablet_bootstrap.cc:492] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: No bootstrap required, opened a new log
I20260812 00:12:15.230711 14338 ts_tablet_manager.cc:1397] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:15.230860 14338 raft_consensus.cc:348] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.230937 14338 raft_consensus.cc:374] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.230957 14338 raft_consensus.cc:729] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94332a6bf34892933db847be7c7119, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.230999 14338 consensus_queue.cc:260] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.231037 14372 ts_tablet_manager.cc:1918] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:15.231092 14372 log.cc:1196] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/d063324fbb3e42d7bc23c789fc3f3141
I20260812 00:12:15.231112 14338 ts_tablet_manager.cc:1428] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:15.231295 14372 ts_tablet_manager.cc:1939] T d063324fbb3e42d7bc23c789fc3f3141 P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting consensus metadata
I20260812 00:12:15.231971 14331 tablet_bootstrap.cc:492] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: No bootstrap required, opened a new log
I20260812 00:12:15.231962 13453 catalog_manager.cc:4841] TS 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609): tablet d063324fbb3e42d7bc23c789fc3f3141 (table test-table [id=ccb4226502fd43f99f63ae2286302fe3]) successfully deleted
I20260812 00:12:15.232023 14331 ts_tablet_manager.cc:1397] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260812 00:12:15.232131 14331 raft_consensus.cc:348] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.232213 14331 raft_consensus.cc:374] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.232240 14331 raft_consensus.cc:729] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.232282 14331 consensus_queue.cc:260] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.232388 14331 ts_tablet_manager.cc:1428] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:15.232645 14334 tablet_bootstrap.cc:492] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: No bootstrap required, opened a new log
I20260812 00:12:15.232694 14334 ts_tablet_manager.cc:1397] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260812 00:12:15.232829 14334 raft_consensus.cc:348] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.232910 14334 raft_consensus.cc:374] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:15.232930 14334 raft_consensus.cc:729] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5457bd7d4c4eef833adfe79ef147d8, State: Initialized, Role: FOLLOWER
I20260812 00:12:15.232973 14334 consensus_queue.cc:260] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.233049 14334 ts_tablet_manager.cc:1428] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:15.361109 14367 raft_consensus.cc:482] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:15.361218 14367 raft_consensus.cc:504] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.361398 14367 leader_election.cc:288] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da94332a6bf34892933db847be7c7119 (127.31.227.2:40253), 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:15.361675 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe13cf30e4f74c429d0428421981430d" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" is_pre_election: true
I20260812 00:12:15.361675 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe13cf30e4f74c429d0428421981430d" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119" is_pre_election: true
I20260812 00:12:15.361797 14206 raft_consensus.cc:2454] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 0.
I20260812 00:12:15.361804 13982 raft_consensus.cc:2454] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 0.
I20260812 00:12:15.361975 13634 leader_election.cc:302] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, 72c480d2608e4363a7884151d6b54ff5; no voters:
I20260812 00:12:15.362088 14367 raft_consensus.cc:2781] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:15.362123 14367 raft_consensus.cc:482] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:15.362139 14367 raft_consensus.cc:3037] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.363034 14367 raft_consensus.cc:504] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.363183 14367 leader_election.cc:288] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 election: Requested vote from peers da94332a6bf34892933db847be7c7119 (127.31.227.2:40253), 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609)
I20260812 00:12:15.363382 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe13cf30e4f74c429d0428421981430d" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da94332a6bf34892933db847be7c7119"
I20260812 00:12:15.363400 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe13cf30e4f74c429d0428421981430d" candidate_uuid: "72c480d2608e4363a7884151d6b54ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5457bd7d4c4eef833adfe79ef147d8"
I20260812 00:12:15.363468 14206 raft_consensus.cc:3037] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.363468 13982 raft_consensus.cc:3037] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:15.364214 13982 raft_consensus.cc:2454] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 1.
I20260812 00:12:15.364214 14206 raft_consensus.cc:2454] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c480d2608e4363a7884151d6b54ff5 in term 1.
I20260812 00:12:15.364434 13634 leader_election.cc:302] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5457bd7d4c4eef833adfe79ef147d8, 72c480d2608e4363a7884151d6b54ff5; no voters:
I20260812 00:12:15.364560 14367 raft_consensus.cc:2781] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:15.364634 14367 raft_consensus.cc:686] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 72c480d2608e4363a7884151d6b54ff5, State: Running, Role: LEADER
I20260812 00:12:15.364693 14367 consensus_queue.cc:237] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } }
I20260812 00:12:15.365345 13477 catalog_manager.cc:5495] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "72c480d2608e4363a7884151d6b54ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c480d2608e4363a7884151d6b54ff5" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 } health_report { overall_health: UNKNOWN } } }
I20260812 00:12:15.376147 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.373734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:43683 (local address 127.31.227.3:34609)
0812 00:12:15.373796 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.373799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.373924 (+ 125us) server_negotiation.cc:407] Connection header received
0812 00:12:15.373993 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.373994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.374005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.374060 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.374165 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.374640 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.375414 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.375620 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.375705 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.375886 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.375927 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.375989 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.376051 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.376073 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:12:15.376171 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.373662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:34609 (local address 127.31.227.1:43683)
0812 00:12:15.373846 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.373858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.373941 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.374064 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.374065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.374067 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:12:15.374118 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.374120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.374730 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.374732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.375328 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.375329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.375708 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.375709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.375766 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.375791 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.375990 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.375993 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.376046 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.376097 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260812 00:12:15.378168 14206 raft_consensus.cc:1261] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Refusing update from remote peer 72c480d2608e4363a7884151d6b54ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.378175 13982 raft_consensus.cc:1261] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Refusing update from remote peer 72c480d2608e4363a7884151d6b54ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:15.378401 14340 consensus_queue.cc:1035] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da94332a6bf34892933db847be7c7119" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.378549 14367 consensus_queue.cc:1035] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5457bd7d4c4eef833adfe79ef147d8" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:15.385620 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.383507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58245 (local address 127.31.227.1:44631)
0812 00:12:15.383549 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:15.383551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:15.383557 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:12:15.383576 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:15.383576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:15.383587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:15.383637 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:15.383766 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.384243 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.385019 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.385198 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.385252 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.385400 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.385424 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.385454 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:15.385494 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 00:12:15.385510 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:12:15.385649 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.383413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44631 (local address 127.31.227.3:58245)
0812 00:12:15.383452 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.383462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:15.383505 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:15.383643 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:15.383644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:15.383646 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:12:15.383692 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.383694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.384309 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.384310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.384949 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:15.384952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:15.385283 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:15.385284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:15.385332 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:15.385336 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:12:15.385461 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:12:15.385462 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:12:15.385492 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 00:12:15.385587 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 00:12:15.392022 13479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:15.392113 13479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:15.392747 13479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d7c3be8d2094b4b835a8f34c8875e17: Sending DeleteTablet for 3 replicas of tablet fe13cf30e4f74c429d0428421981430d
I20260812 00:12:15.393024 13948 tablet_service.cc:1514] Processing DeleteTablet for tablet fe13cf30e4f74c429d0428421981430d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260812 00:12:15.393067 14181 tablet_service.cc:1514] Processing DeleteTablet for tablet fe13cf30e4f74c429d0428421981430d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260812 00:12:15.393144 14374 tablet_replica.cc:330] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: stopping tablet replica
I20260812 00:12:15.393200 14374 raft_consensus.cc:2229] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:15.393245 14374 raft_consensus.cc:2258] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.393270 14372 tablet_replica.cc:330] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: stopping tablet replica
I20260812 00:12:15.393334 14372 raft_consensus.cc:2229] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:15.393376 14372 raft_consensus.cc:2258] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.393421 13722 tablet_service.cc:1514] Processing DeleteTablet for tablet fe13cf30e4f74c429d0428421981430d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:15 UTC) from {username='oryx', principal='oryx/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:40514
I20260812 00:12:15.393543 14373 tablet_replica.cc:330] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: stopping tablet replica
I20260812 00:12:15.393602 14373 raft_consensus.cc:2229] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:15.393640 14374 ts_tablet_manager.cc:1905] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.393662 14373 raft_consensus.cc:2258] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:15.393980 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 13606
I20260812 00:12:15.394057 14373 ts_tablet_manager.cc:1905] T fe13cf30e4f74c429d0428421981430d P 72c480d2608e4363a7884151d6b54ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.395404 14374 ts_tablet_manager.cc:1918] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:12:15.395457 14374 log.cc:1196] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/fe13cf30e4f74c429d0428421981430d
I20260812 00:12:15.395650 14374 ts_tablet_manager.cc:1939] T fe13cf30e4f74c429d0428421981430d P da94332a6bf34892933db847be7c7119: Deleting consensus metadata
I20260812 00:12:15.397174 13452 catalog_manager.cc:4841] TS da94332a6bf34892933db847be7c7119 (127.31.227.2:40253): tablet fe13cf30e4f74c429d0428421981430d (table test-table [id=9caa149ff8c04fa78e037a1421f46798]) successfully deleted
I20260812 00:12:15.405484 14372 ts_tablet_manager.cc:1905] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:15.407333 14372 ts_tablet_manager.cc:1918] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:12:15.407433 14372 log.cc:1196] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/fe13cf30e4f74c429d0428421981430d
I20260812 00:12:15.407652 14372 ts_tablet_manager.cc:1939] T fe13cf30e4f74c429d0428421981430d P 2d5457bd7d4c4eef833adfe79ef147d8: Deleting consensus metadata
I20260812 00:12:15.408025 13453 catalog_manager.cc:4841] TS 2d5457bd7d4c4eef833adfe79ef147d8 (127.31.227.3:34609): tablet fe13cf30e4f74c429d0428421981430d (table test-table [id=9caa149ff8c04fa78e037a1421f46798]) successfully deleted
W20260812 00:12:15.412240 13454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:44631 (error 108)
I20260812 00:12:15.412511 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 13846
I20260812 00:12:15.412864 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:15.412684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44631 (local address 127.0.0.1:40530)
0812 00:12:15.412778 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:15.412817 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:44631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":37,"threads_started":1}
W20260812 00:12:15.412952 13454 catalog_manager.cc:4584] TS 72c480d2608e4363a7884151d6b54ff5 (127.31.227.1:44631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe13cf30e4f74c429d0428421981430d: Network error: Client connection negotiation failed: client connection to 127.31.227.1:44631: connect: Connection refused (error 111)
I20260812 00:12:15.429924 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 14080
I20260812 00:12:15.451781 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 13425
2026-08-12T00:12:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2771 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[14486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[14486](info): set up 2 sockets
Aug 12 00:12:15 dist-test-slave-zr1t krb5kdc[14486](info): commencing operation
krb5kdc: starting...
W20260812 00:12:16.738286 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:16 dist-test-slave-zr1t krb5kdc[14486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:12:16.864589 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46779
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:16.948832 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:16.949103 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:16.949146 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:16.950397 14504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:16.950474 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:16.950500 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:16.950521 14504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:16.950551 14504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:16.952067 14504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:16.952376 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:16.952622 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:12:16.957767 14504 server_base.cc:1034] running on GCE node
W20260812 00:12:16.957718 14512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:16.957715 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:16.957719 14511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:16.958494 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:16.958832 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:16.959996 14504 hybrid_clock.cc:648] HybridClock initialized: now 1786493536959977 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T00:12:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1510 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:17 dist-test-slave-zr1t krb5kdc[14521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:17 dist-test-slave-zr1t krb5kdc[14521](info): set up 2 sockets
Aug 12 00:12:17 dist-test-slave-zr1t krb5kdc[14521](info): commencing operation
krb5kdc: starting...
W20260812 00:12:18.244362 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:12:18.362903 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46721
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:18.446830 14545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:18.447023 14545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:18.447045 14545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:18.448221 14545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:18.448262 14545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:18.448277 14545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:18.448290 14545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:18.448302 14545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:18.449820 14545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:18.450062 14545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:18.450281 14545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:18.455188 14561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.455193 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:18.455474 14545 server_base.cc:1034] running on GCE node
W20260812 00:12:18.455229 14557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:18.455770 14545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:18.456058 14545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:18.457238 14545 hybrid_clock.cc:648] HybridClock initialized: now 1786493538457194 us; error 60 us; skew 500 ppm
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:18.459935 14545 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:12:18.460327 14545 webserver.cc:460] Webserver started at http://127.31.227.62:32959/ using document root <none> and password file <none>
I20260812 00:12:18.460542 14545 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:18.460587 14545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:18.460677 14545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:18.461797 14545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "a74dd3b0dc3049ce9708c523afb4507a"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "044d378766918a26c2a09c72a99887a8"
server_key_iv: "bc937308deccbc3520aefab2205ef641"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.462118 14545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "a74dd3b0dc3049ce9708c523afb4507a"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "044d378766918a26c2a09c72a99887a8"
server_key_iv: "bc937308deccbc3520aefab2205ef641"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.463475 14545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.464191 14568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.464376 14545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.464473 14545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "a74dd3b0dc3049ce9708c523afb4507a"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "044d378766918a26c2a09c72a99887a8"
server_key_iv: "bc937308deccbc3520aefab2205ef641"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.464557 14545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:18.485210 14545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:18.487972 14545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:18.488090 14545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:18.492110 14545 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:46721
I20260812 00:12:18.492158 14656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:46721 every 8 connection(s)
I20260812 00:12:18.492542 14545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:18.493485 14657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:18.495931 14657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Bootstrap starting.
I20260812 00:12:18.496477 14657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:18.496712 14657 log.cc:824] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:18.497418 14657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: No bootstrap required, opened a new log
I20260812 00:12:18.498503 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 14545
I20260812 00:12:18.498643 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:18.498854 32652 external_mini_cluster.cc:1428] Setting key 2e671dad4cbba00ce88ab65883b2ad82
I20260812 00:12:18.499697 14657 raft_consensus.cc:348] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:18.499847 14657 raft_consensus.cc:374] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:18.499876 14657 raft_consensus.cc:729] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74dd3b0dc3049ce9708c523afb4507a, State: Initialized, Role: FOLLOWER
I20260812 00:12:18.500030 14657 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:18.500124 14657 raft_consensus.cc:388] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:18.500173 14657 raft_consensus.cc:482] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:18.500222 14657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:18.501390 14657 raft_consensus.cc:504] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:18.501487 14657 leader_election.cc:302] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a74dd3b0dc3049ce9708c523afb4507a; no voters:
I20260812 00:12:18.501748 14657 leader_election.cc:288] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:18.501768 14666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:18.501885 14666 raft_consensus.cc:686] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 LEADER]: Becoming Leader. State: Replica: a74dd3b0dc3049ce9708c523afb4507a, State: Running, Role: LEADER
I20260812 00:12:18.501977 14666 consensus_queue.cc:237] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:18.502060 14657 sys_catalog.cc:564] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:18.502337 14666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a74dd3b0dc3049ce9708c523afb4507a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } } }
I20260812 00:12:18.502420 14666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:18.503007 14675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:18.503409 14668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74dd3b0dc3049ce9708c523afb4507a. Latest consensus state: current_term: 1 leader_uuid: "a74dd3b0dc3049ce9708c523afb4507a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } } }
I20260812 00:12:18.503597 14668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:18.503773 14675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:18.505658 14675 catalog_manager.cc:1347] Generated new cluster ID: 87ab1336437145598b8ae9d58af0cc6e
I20260812 00:12:18.505717 14675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:18.510066 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:18.499442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.31.227.62:46721)
0812 00:12:18.499671 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:18.499680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:18.499707 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:12:18.499816 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:18.499822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:18.499890 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:18.500090 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:18.500952 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.502156 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.503254 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.503868 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.504381 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:18.506743 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:18.506747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:18.506747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:18.506760 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:18.508439 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:18.508818 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:18.508821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:18.508822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:18.508948 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:18.509171 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:18.509172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:18.509173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:18.509410 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:18.509609 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:18.509721 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:12:18.509863 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":134,"threads_started":1}
I20260812 00:12:18.514592 14675 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:18.515270 14675 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:18.523779 14675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Generated new TSK 0
I20260812 00:12:18.523959 14675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:12:18.555020 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46721
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:18.639004 14705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:18.639204 14705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:18.639227 14705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:18.640363 14705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:18.640403 14705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:18.640427 14705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:18.641923 14705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:18.642171 14705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:18.642421 14705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:18.647480 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.647495 14711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.647521 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:18.647646 14705 server_base.cc:1034] running on GCE node
I20260812 00:12:18.647861 14705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:18.648058 14705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:18.649197 14705 hybrid_clock.cc:648] HybridClock initialized: now 1786493538649184 us; error 28 us; skew 500 ppm
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:18.651714 14705 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:12:18.652170 14705 webserver.cc:460] Webserver started at http://127.31.227.1:33021/ using document root <none> and password file <none>
I20260812 00:12:18.652395 14705 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:18.652441 14705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:18.652570 14705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:18.653657 14705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "f809b6baf7ce43fba00412f9e6d832d5"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "2d26bb4b33a50817dc51c970138cabf3"
server_key_iv: "d5d9b392d8786f638006cbd5fd620ef7"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.654027 14705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "f809b6baf7ce43fba00412f9e6d832d5"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "2d26bb4b33a50817dc51c970138cabf3"
server_key_iv: "d5d9b392d8786f638006cbd5fd620ef7"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.655377 14705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.656155 14731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.656334 14705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.656445 14705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "f809b6baf7ce43fba00412f9e6d832d5"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "2d26bb4b33a50817dc51c970138cabf3"
server_key_iv: "d5d9b392d8786f638006cbd5fd620ef7"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.656523 14705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:18.672046 14705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:18.674044 14705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:18.674206 14705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:18.674451 14705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:18.674774 14705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:18.674831 14705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.674875 14705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:18.674899 14705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.680958 14705 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:46273
I20260812 00:12:18.681047 14933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:46273 every 8 connection(s)
I20260812 00:12:18.681385 14705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:18.690320 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 14705
I20260812 00:12:18.690454 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:18.690691 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:18.682168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:48505 (local address 127.31.227.62:46721)
0812 00:12:18.682235 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:18.682237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:18.682505 (+ 268us) server_negotiation.cc:407] Connection header received
0812 00:12:18.683048 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:18.683049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:18.683066 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:18.683124 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:18.684096 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.684605 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.685635 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.685857 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.685944 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:18.688125 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:18.688127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:18.688127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:18.688135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:18.689318 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:18.689742 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:18.689743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:18.689743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:18.689795 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:18.690015 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:18.690016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:18.690016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:18.690111 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:18.690159 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:18.690534 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 00:12:18.690627 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:12:18.690696 32652 external_mini_cluster.cc:1428] Setting key 070c9161198f223df67be35a39a681d9
I20260812 00:12:18.690867 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:18.682152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:48505)
0812 00:12:18.682347 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:18.682406 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:18.682934 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:18.683198 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:18.683202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:18.683782 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:18.683974 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:18.683979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.684704 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.684705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:18.685508 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:18.685510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.685957 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.685958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:18.686075 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:18.686611 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:18.686620 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:18.688036 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:18.689393 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:18.689401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:18.689403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:18.689688 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:18.689866 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:18.689867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:18.689867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:18.689970 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:18.690174 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:18.690181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:18.690348 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 00:12:18.690471 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 00:12:18.690647 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260812 00:12:18.693087 14934 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46721
I20260812 00:12:18.693172 14934 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:18.693364 14934 heartbeater.cc:511] Master 127.31.227.62:46721 requested a full tablet report, sending...
I20260812 00:12:18.694016 14594 ts_manager.cc:194] Registered new tserver with Master: f809b6baf7ce43fba00412f9e6d832d5 (127.31.227.1:46273)
I20260812 00:12:18.694983 14594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:48505
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:12:18.718138 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46721
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:18.804252 14941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:18.804461 14941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:18.804484 14941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:18.805598 14941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:18.805630 14941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:18.805656 14941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:18.807015 14941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:18.807204 14941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:18.807389 14941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:18.812354 14947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.812511 14949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.812364 14951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:18.812578 14941 server_base.cc:1034] running on GCE node
I20260812 00:12:18.812779 14941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:18.813027 14941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:18.814218 14941 hybrid_clock.cc:648] HybridClock initialized: now 1786493538814203 us; error 46 us; skew 500 ppm
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:18.816759 14941 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:12:18.817210 14941 webserver.cc:460] Webserver started at http://127.31.227.2:38097/ using document root <none> and password file <none>
I20260812 00:12:18.817414 14941 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:18.817456 14941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:18.817606 14941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:18.818634 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "f94be57808ef43e9bce488cc3dea9d13"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "7b6f7b6d306b4165672dd818ba8fe42a"
server_key_iv: "0c31e54674e8d422242f98aaf4ba99ac"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.818934 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "f94be57808ef43e9bce488cc3dea9d13"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "7b6f7b6d306b4165672dd818ba8fe42a"
server_key_iv: "0c31e54674e8d422242f98aaf4ba99ac"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.820191 14941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:18.820936 14963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.821161 14941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.821274 14941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "f94be57808ef43e9bce488cc3dea9d13"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "7b6f7b6d306b4165672dd818ba8fe42a"
server_key_iv: "0c31e54674e8d422242f98aaf4ba99ac"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.821597 14941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:18.836153 14941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:18.838604 14941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:18.838770 14941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:18.839006 14941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:18.839330 14941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:18.839365 14941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.839386 14941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:18.839433 14941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.845664 14941 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:43703
I20260812 00:12:18.845758 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:43703 every 8 connection(s)
I20260812 00:12:18.846066 14941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:18.853669 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 14941
I20260812 00:12:18.853802 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:18.854046 32652 external_mini_cluster.cc:1428] Setting key 514551471a416b4f4d07f23290a5ce00
I20260812 00:12:18.855654 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:18.846881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34515 (local address 127.31.227.62:46721)
0812 00:12:18.846936 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:18.846939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:18.847169 (+ 230us) server_negotiation.cc:407] Connection header received
0812 00:12:18.847646 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:18.847647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:18.847658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:18.847711 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:18.848608 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.849097 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.850134 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.850320 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.850376 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:18.852598 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:18.852600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:18.852601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:18.852607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:18.854065 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:18.854568 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:18.854570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:18.854570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:18.854638 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:18.854916 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:18.854917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:18.854918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:18.855072 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:18.855128 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:18.855495 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 00:12:18.855590 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:12:18.855926 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:18.846860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:34515)
0812 00:12:18.847048 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:18.847084 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:18.847562 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:18.847764 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:18.847767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:18.848336 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:18.848534 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:18.848538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.849178 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.849178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:18.850059 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:18.850062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:18.850395 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:18.850395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:18.850493 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:18.851029 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:18.851037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:18.852487 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:18.854171 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:18.854179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:18.854182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:18.854494 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:18.854711 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:18.854712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:18.854712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:18.854841 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:18.855142 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:18.855150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:18.855352 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 00:12:18.855472 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:12:18.855654 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260812 00:12:18.857333 15167 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46721
I20260812 00:12:18.857417 15167 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:18.857643 15167 heartbeater.cc:511] Master 127.31.227.62:46721 requested a full tablet report, sending...
I20260812 00:12:18.858108 14594 ts_manager.cc:194] Registered new tserver with Master: f94be57808ef43e9bce488cc3dea9d13 (127.31.227.2:43703)
I20260812 00:12:18.858624 14594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:34515
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:12:18.880774 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46721
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:18.966001 15173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:18.966209 15173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:18.966235 15173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:18.967411 15173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:18.967445 15173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:18.967470 15173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:18.968848 15173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:18.969051 15173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:18.969240 15173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:18.974453 15186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.974498 15181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:18.974581 15184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:18.974779 15173 server_base.cc:1034] running on GCE node
I20260812 00:12:18.974996 15173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:18.975193 15173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:18.976363 15173 hybrid_clock.cc:648] HybridClock initialized: now 1786493538976356 us; error 37 us; skew 500 ppm
Aug 12 00:12:18 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:18.979044 15173 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:12:18.979478 15173 webserver.cc:460] Webserver started at http://127.31.227.3:44911/ using document root <none> and password file <none>
I20260812 00:12:18.979724 15173 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:18.979794 15173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:18.979947 15173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:18.981027 15173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "4d8111e8bffb4ebf988dc1459cd8989d"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "677974cafde11c2da1b2bb40d6c4b7fc"
server_key_iv: "5dbc13faeca1af435c5f66c015862f05"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.981406 15173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "4d8111e8bffb4ebf988dc1459cd8989d"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "677974cafde11c2da1b2bb40d6c4b7fc"
server_key_iv: "5dbc13faeca1af435c5f66c015862f05"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.982832 15173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.983637 15193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:18.983839 15173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:18.983947 15173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "4d8111e8bffb4ebf988dc1459cd8989d"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "677974cafde11c2da1b2bb40d6c4b7fc"
server_key_iv: "5dbc13faeca1af435c5f66c015862f05"
server_key_version: "encryptionkey@0"
I20260812 00:12:18.984023 15173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:18.997450 15173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:18.999881 15173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:19.000053 15173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:19.000296 15173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:19.000629 15173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:19.000684 15173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.000726 15173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:19.000754 15173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.007344 15173 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:42077
I20260812 00:12:19.007380 15393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:42077 every 8 connection(s)
I20260812 00:12:19.007730 15173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493538, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:19.016746 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 15173
I20260812 00:12:19.016875 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:19.016986 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.008487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:44895 (local address 127.31.227.62:46721)
0812 00:12:19.008560 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.008562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.008786 (+ 224us) server_negotiation.cc:407] Connection header received
0812 00:12:19.009309 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.009310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.009322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.009406 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.010426 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.010924 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.011909 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.012103 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.012169 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.014402 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:19.014404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:19.014404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:19.014411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:19.015716 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:19.016093 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:19.016094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:19.016094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:19.016148 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:19.016364 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:19.016365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:19.016365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:19.016471 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:19.016525 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:19.016778 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 00:12:19.016880 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:12:19.017072 32652 external_mini_cluster.cc:1428] Setting key 4d535ee0d7cb36078b98916afcee9dd6
I20260812 00:12:19.017122 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.008481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:44895)
0812 00:12:19.008642 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.008679 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.009187 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.009462 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.009465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.010138 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.010341 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.010347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.011009 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.011010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.011806 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.011817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.012188 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.012189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.012288 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.012845 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.012854 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.014310 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:19.015791 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:19.015797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:19.015799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:19.016024 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:19.016210 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:19.016210 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:19.016210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:19.016305 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:19.016528 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:19.016533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:19.016643 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 00:12:19.016724 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:12:19.016891 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260812 00:12:19.017611 15395 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46721
I20260812 00:12:19.017717 15395 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:19.017918 15395 heartbeater.cc:511] Master 127.31.227.62:46721 requested a full tablet report, sending...
I20260812 00:12:19.018361 14595 ts_manager.cc:194] Registered new tserver with Master: 4d8111e8bffb4ebf988dc1459cd8989d (127.31.227.3:42077)
I20260812 00:12:19.018749 14595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:44895
I20260812 00:12:19.028867 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab.
Entry for principal oryx/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab.
WARNING: no policy specified for oryx/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab.
Entry for principal oryx/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab.
WARNING: no policy specified for oryx/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab.
Entry for principal oryx/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab.
I20260812 00:12:19.107527 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 14705
I20260812 00:12:19.120676 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 14941
I20260812 00:12:19.136627 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 15173
I20260812 00:12:19.154090 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 14545
I20260812 00:12:19.164516 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46721
--webserver_interface=127.31.227.62
--webserver_port=32959
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:19.248313 15406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:19.248528 15406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:19.248551 15406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:19.249681 15406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:19.249719 15406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:19.249733 15406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:19.249745 15406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:19.249758 15406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:19.251196 15406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=32959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:19.251401 15406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:19.251602 15406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:19.256629 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:19.256636 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:19.256629 15418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:19.256984 15406 server_base.cc:1034] running on GCE node
I20260812 00:12:19.257283 15406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:19.257669 15406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:19.258817 15406 hybrid_clock.cc:648] HybridClock initialized: now 1786493539258807 us; error 29 us; skew 500 ppm
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493539, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:19.265923 15406 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:12:19.266422 15406 webserver.cc:460] Webserver started at http://127.31.227.62:32959/ using document root <none> and password file <none>
I20260812 00:12:19.266615 15406 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:19.266670 15406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:19.268009 15406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:19.268652 15428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.268827 15406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.268951 15406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "a74dd3b0dc3049ce9708c523afb4507a"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "044d378766918a26c2a09c72a99887a8"
server_key_iv: "bc937308deccbc3520aefab2205ef641"
server_key_version: "encryptionkey@0"
I20260812 00:12:19.269320 15406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:19.283365 15406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:19.285175 15406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:19.285346 15406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:19.290095 15406 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:46721
I20260812 00:12:19.290191 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:46721 every 8 connection(s)
I20260812 00:12:19.290541 15406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:19.292114 15513 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:12:19.292881 15513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Bootstrap starting.
I20260812 00:12:19.300333 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 15406
I20260812 00:12:19.300753 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:46273
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:46721
--webserver_port=33021
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:19.303467 15513 log.cc:824] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:19.305197 15513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:19.305568 15513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Bootstrap complete.
I20260812 00:12:19.308126 15513 raft_consensus.cc:348] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:19.308367 15513 raft_consensus.cc:729] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74dd3b0dc3049ce9708c523afb4507a, State: Initialized, Role: FOLLOWER
I20260812 00:12:19.308552 15513 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:19.308627 15513 raft_consensus.cc:388] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:19.308696 15513 raft_consensus.cc:482] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:19.308749 15513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:12:19.309950 15513 raft_consensus.cc:504] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:19.310065 15513 leader_election.cc:302] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a74dd3b0dc3049ce9708c523afb4507a; no voters:
I20260812 00:12:19.310297 15513 leader_election.cc:288] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:12:19.310343 15518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:12:19.310523 15518 raft_consensus.cc:686] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [term 2 LEADER]: Becoming Leader. State: Replica: a74dd3b0dc3049ce9708c523afb4507a, State: Running, Role: LEADER
I20260812 00:12:19.310604 15513 sys_catalog.cc:564] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:19.310736 15518 consensus_queue.cc:237] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } }
I20260812 00:12:19.311102 15521 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a74dd3b0dc3049ce9708c523afb4507a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } } }
I20260812 00:12:19.311240 15521 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:19.311100 15518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74dd3b0dc3049ce9708c523afb4507a. Latest consensus state: current_term: 2 leader_uuid: "a74dd3b0dc3049ce9708c523afb4507a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74dd3b0dc3049ce9708c523afb4507a" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46721 } } }
I20260812 00:12:19.311894 15518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:19.312006 15530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:19.312816 15530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:19.312958 15530 catalog_manager.cc:1259] Loaded cluster ID: 87ab1336437145598b8ae9d58af0cc6e
I20260812 00:12:19.313001 15530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:19.314275 15530 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:19.314478 15530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a74dd3b0dc3049ce9708c523afb4507a: Loaded TSK: 0
I20260812 00:12:19.314708 15530 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:12:19.387310 15515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:19.387554 15515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:19.387595 15515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:19.388787 15515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:19.388866 15515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:19.388916 15515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:19.390544 15515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.1.keytab
--principal=oryx/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:46273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=33021
--tserver_master_addrs=127.31.227.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:19.390823 15515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:19.391084 15515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:19.397333 15551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:19.397578 15515 server_base.cc:1034] running on GCE node
W20260812 00:12:19.397360 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:19.397333 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:19.397948 15515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:19.398267 15515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:19.399463 15515 hybrid_clock.cc:648] HybridClock initialized: now 1786493539399436 us; error 41 us; skew 500 ppm
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493539, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:19.402725 15515 init.cc:377] Logged in from keytab as oryx/127.31.227.1@KRBTEST.COM (short username oryx)
I20260812 00:12:19.403280 15515 webserver.cc:460] Webserver started at http://127.31.227.1:33021/ using document root <none> and password file <none>
I20260812 00:12:19.403558 15515 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:19.403640 15515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:19.405436 15515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:19.406209 15563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.406407 15515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:19.406520 15515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "f809b6baf7ce43fba00412f9e6d832d5"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "2d26bb4b33a50817dc51c970138cabf3"
server_key_iv: "d5d9b392d8786f638006cbd5fd620ef7"
server_key_version: "encryptionkey@0"
I20260812 00:12:19.406941 15515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:19.436865 15515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:19.439419 15515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:19.439572 15515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:19.439857 15515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:19.440194 15515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:19.440251 15515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.440287 15515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:19.440315 15515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.446630 15515 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:46273
I20260812 00:12:19.446686 15750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:46273 every 8 connection(s)
I20260812 00:12:19.447050 15515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:19.447510 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 15515
I20260812 00:12:19.447863 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:43703
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:46721
--webserver_port=38097
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:19.457742 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.447840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:36529)
0812 00:12:19.448258 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.448309 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.449091 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.449583 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.449588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.450245 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.450452 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.450457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.451723 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.451724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.453167 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.453171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.453847 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.453849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.454010 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.454572 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.454584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.457460 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":124,"threads_started":1}
W20260812 00:12:19.457806 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.447959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:36529 (local address 127.31.227.62:46721)
0812 00:12:19.448229 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.448239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.448400 (+ 161us) server_negotiation.cc:407] Connection header received
0812 00:12:19.449291 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.449297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.449357 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.449468 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.450592 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.451608 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.453275 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.453752 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.453835 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.457659 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:36529: BlockingRecv error: recv got EOF from 127.31.227.1:36529 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
W20260812 00:12:19.458017 15751 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:46721 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.227.62:46721: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:19.463384 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.458296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:47467)
0812 00:12:19.458379 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.458391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.458446 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.458585 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.458586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.458706 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.458752 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.458753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.459785 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.459786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.460289 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.460291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.460903 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.460905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.460960 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.461443 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.461448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.463333 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 00:12:19.463512 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.458332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:47467 (local address 127.31.227.62:46721)
0812 00:12:19.458373 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.458375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.458437 (+ 62us) server_negotiation.cc:407] Connection header received
0812 00:12:19.458491 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.458492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.458507 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.458578 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.458828 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.459640 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.460476 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.460828 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.460902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.463447 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:47467: BlockingRecv error: recv got EOF from 127.31.227.1:47467 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:19.468279 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.463889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:52623)
0812 00:12:19.463954 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.463965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.464005 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.464150 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.464151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.464257 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.464284 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.464285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.465148 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.465149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.465711 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.465713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.466149 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.466150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.466191 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.466626 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.466631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.468234 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 00:12:19.468403 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.463949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:52623 (local address 127.31.227.62:46721)
0812 00:12:19.463983 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.463985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.463994 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:19.464069 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.464070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.464080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.464140 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.464348 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.465074 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.465781 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.466076 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.466144 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.468336 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:52623: BlockingRecv error: recv got EOF from 127.31.227.1:52623 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 00:12:19.468863 15751 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:12:19.537235 15758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:19.537431 15758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:19.537456 15758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:19.538632 15758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:19.538676 15758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:19.538702 15758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:19.540084 15758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.2.keytab
--principal=oryx/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:43703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=38097
--tserver_master_addrs=127.31.227.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:19.540269 15758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:19.540437 15758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:19.545398 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:19.545585 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:19.545626 15758 server_base.cc:1034] running on GCE node
W20260812 00:12:19.545400 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:19.545881 15758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:19.546118 15758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:19.547274 15758 hybrid_clock.cc:648] HybridClock initialized: now 1786493539547249 us; error 42 us; skew 500 ppm
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493539, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:19.549773 15758 init.cc:377] Logged in from keytab as oryx/127.31.227.2@KRBTEST.COM (short username oryx)
I20260812 00:12:19.550148 15758 webserver.cc:460] Webserver started at http://127.31.227.2:38097/ using document root <none> and password file <none>
I20260812 00:12:19.550356 15758 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:19.550410 15758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:19.551687 15758 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:19.552232 15788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.552423 15758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:19.552507 15758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "f94be57808ef43e9bce488cc3dea9d13"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "7b6f7b6d306b4165672dd818ba8fe42a"
server_key_iv: "0c31e54674e8d422242f98aaf4ba99ac"
server_key_version: "encryptionkey@0"
I20260812 00:12:19.552809 15758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:19.582412 15758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:19.584266 15758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:19.584373 15758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:19.584569 15758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:19.584812 15758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:19.584843 15758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.584868 15758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:19.584882 15758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.590965 15758 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:43703
I20260812 00:12:19.591049 16006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:43703 every 8 connection(s)
I20260812 00:12:19.591372 15758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:19.594453 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 15758
I20260812 00:12:19.594764 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:42077
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:46721
--webserver_port=44911
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:12:19.599462 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.592127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:33757 (local address 127.31.227.62:46721)
0812 00:12:19.592184 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.592186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.592389 (+ 203us) server_negotiation.cc:407] Connection header received
0812 00:12:19.592881 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.592882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.592908 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.592955 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.593934 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.594483 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.595595 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.595791 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.595878 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.599406 (+ 3528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:33757: BlockingRecv error: recv got EOF from 127.31.227.2:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:12:19.599514 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.592079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:33757)
0812 00:12:19.592261 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.592303 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.592773 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.593038 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.593042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.593647 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.593847 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.593852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.594582 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.594584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.595493 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.595496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.595894 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.595895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.596022 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.596543 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.596556 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.599313 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
W20260812 00:12:19.599752 16008 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:46721 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.227.62:46721: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:12:19.604828 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.600088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:41775 (local address 127.31.227.62:46721)
0812 00:12:19.600154 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.600156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.600165 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:19.600190 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.600191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.600200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.600272 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.600542 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.601011 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.601723 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.601927 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.601999 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.604782 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:41775: BlockingRecv error: recv got EOF from 127.31.227.2:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:12:19.604822 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.600020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:41775)
0812 00:12:19.600055 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.600066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.600121 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.600276 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.600277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.600414 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.600461 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.600462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.601093 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.601094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.601643 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.601645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.602011 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.602012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.602081 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.602589 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.602594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.604702 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:19.610536 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.605225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:34307)
0812 00:12:19.605335 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.605345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.605382 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.605585 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.605586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.605695 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.605723 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.605724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.606778 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.606780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.607263 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.607264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.607863 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.607865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.607915 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.608445 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.608449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.610458 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260812 00:12:19.610657 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.605287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34307 (local address 127.31.227.62:46721)
0812 00:12:19.605362 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.605365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.605377 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:12:19.605437 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.605438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.605451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.605558 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.605813 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.606669 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.607357 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.607746 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.607845 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.610547 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:34307: BlockingRecv error: recv got EOF from 127.31.227.2:34307 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 00:12:19.610702 16008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:12:19.707404 16011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:19.707873 16011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:19.707921 16011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:19.710039 16011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:19.710132 16011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:19.710175 16011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:19.713110 16011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/krb5kdc/oryx_127.31.227.3.keytab
--principal=oryx/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=44911
--tserver_master_addrs=127.31.227.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:19.713440 16011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:19.713851 16011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:19.722500 16023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:19.722560 16021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:19.722543 16025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:19.723459 16011 server_base.cc:1034] running on GCE node
I20260812 00:12:19.723910 16011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:19.724345 16011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:19.725610 16011 hybrid_clock.cc:648] HybridClock initialized: now 1786493539725568 us; error 44 us; skew 500 ppm
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493539, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:19.730567 16011 init.cc:377] Logged in from keytab as oryx/127.31.227.3@KRBTEST.COM (short username oryx)
I20260812 00:12:19.731259 16011 webserver.cc:460] Webserver started at http://127.31.227.3:44911/ using document root <none> and password file <none>
I20260812 00:12:19.731663 16011 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:19.731796 16011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:19.734953 16011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 00:12:19.737360 16038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.737752 16011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.002s
I20260812 00:12:19.737987 16011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "4d8111e8bffb4ebf988dc1459cd8989d"
format_stamp: "Formatted at 2026-08-12 00:12:18 on dist-test-slave-zr1t"
server_key: "677974cafde11c2da1b2bb40d6c4b7fc"
server_key_iv: "5dbc13faeca1af435c5f66c015862f05"
server_key_version: "encryptionkey@0"
I20260812 00:12:19.738729 16011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:19.758575 16011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:19.762112 16011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:19.762454 16011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:19.762935 16011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:19.763552 16011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:19.763633 16011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.763749 16011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:19.763813 16011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:19.776067 16011 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:42077
I20260812 00:12:19.776196 16234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:42077 every 8 connection(s)
I20260812 00:12:19.776834 16011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:19.783059 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 16011
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:12:19.791047 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.778189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:39565 (local address 127.31.227.62:46721)
0812 00:12:19.778288 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.778295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.778737 (+ 442us) server_negotiation.cc:407] Connection header received
0812 00:12:19.779633 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.779635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.779647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.779728 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.781815 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.782575 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.785007 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.785334 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.785627 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.790983 (+ 5356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:39565: BlockingRecv error: recv got EOF from 127.31.227.3:39565 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 00:12:19.791289 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.778228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:39565)
0812 00:12:19.778543 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.778600 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.779487 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.779832 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.779843 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.780993 (+ 1150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.781432 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.781439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.782770 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.782772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.784783 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.784786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.785617 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.785619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.785843 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.786830 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.786848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.790903 (+ 4055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":168,"threads_started":1}
W20260812 00:12:19.791786 16235 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:46721 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.227.62:46721: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
I20260812 00:12:19.797783 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.784099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.31.227.62:46721)
0812 00:12:19.784386 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.784389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.784426 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:12:19.784501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.784503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.784539 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.784754 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.786001 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.787109 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.788373 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.788697 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.788831 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.791468 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:19.791475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:19.791476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:19.791509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:19.795053 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:19.795696 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:19.795707 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:19.795708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:19.795875 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:19.796244 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:19.796254 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:19.796255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:19.796845 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:19.797145 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:19.797284 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 00:12:19.797474 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":149,"threads_started":1}
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:19.800226 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.792265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:48029)
0812 00:12:19.792339 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.792349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.792421 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.792726 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.792727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.792989 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.793083 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.793086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.794217 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.794219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.795467 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.795470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.796137 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.796138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.796271 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.796828 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.796833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.800151 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 00:12:19.800328 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.792526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:48029 (local address 127.31.227.62:46721)
0812 00:12:19.792580 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.792582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.792605 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:12:19.792631 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.792632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.792641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.792695 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.793288 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.794082 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.795582 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.795961 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.796091 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.800281 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:48029: BlockingRecv error: recv got EOF from 127.31.227.3:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:19 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:19.807474 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:19.800674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:46009)
0812 00:12:19.800763 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:19.800784 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:19.800875 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:19.801122 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:19.801132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:19.801425 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:19.801467 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.801469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.802590 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.802591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.803323 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:19.803325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.803981 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.803982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:19.804115 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.804742 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:19.804755 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:19.807381 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 00:12:19.807511 15755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:19.800785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:46009 (local address 127.31.227.62:46721)
0812 00:12:19.800878 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:19.800881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:19.800889 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:19.800973 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:19.800974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:19.801001 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:19.801119 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:19.801671 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.802397 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.803498 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:19.803793 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:19.803993 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:19.807449 (+ 3456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:46009: BlockingRecv error: recv got EOF from 127.31.227.3:46009 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 00:12:19.807652 16235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 00:12:20 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:20 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:20.477766 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:20.469469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:56469)
0812 00:12:20.469737 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:20.469752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:20.469859 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:20.470094 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:20.470095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:20.470383 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:20.470463 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:20.470465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.471569 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.471570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:20.472687 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:20.472698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.473296 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.473296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:20.473378 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:20.474312 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:20.474318 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:20.477643 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":154,"threads_started":1}
W20260812 00:12:20.477819 16243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:20.469606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:56469 (local address 127.31.227.62:46721)
0812 00:12:20.469810 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:20.469812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:20.469834 (+ 22us) server_negotiation.cc:407] Connection header received
0812 00:12:20.469954 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:20.469954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:20.469976 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:20.470077 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:20.470562 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.471378 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.472835 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.473210 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.473292 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:20.477732 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:56469: BlockingRecv error: recv got EOF from 127.31.227.1:56469 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
Aug 12 00:12:20 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:20 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:20.618513 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:20.611239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:43987)
0812 00:12:20.611464 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:20.611485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:20.611620 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:20.611955 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:20.611956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:20.612232 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:20.612313 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:20.612315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.613214 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.613215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:20.613988 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:20.613999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.614507 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.614509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:20.614569 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:20.615406 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:20.615415 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:20.618399 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":137,"threads_started":1}
W20260812 00:12:20.618630 16243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:20.611304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:43987 (local address 127.31.227.62:46721)
0812 00:12:20.611418 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:20.611421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:20.611578 (+ 157us) server_negotiation.cc:407] Connection header received
0812 00:12:20.611801 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:20.611802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:20.611816 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:20.611944 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:20.612414 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.613127 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.614124 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.614418 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.614507 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:20.618548 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:43987: BlockingRecv error: recv got EOF from 127.31.227.2:43987 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 00:12:20 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:20 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:20.816133 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:20.808240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:46575)
0812 00:12:20.808429 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:20.808449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:20.808531 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:20.808790 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:20.808791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:20.809128 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:20.809208 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:20.809211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.810287 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.810289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:20.811290 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:20.811293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.811905 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.811907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:20.811994 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:20.812782 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:20.812790 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:20.815928 (+ 3138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
W20260812 00:12:20.816196 16243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:20.808378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:46575 (local address 127.31.227.62:46721)
0812 00:12:20.808474 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:20.808476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:20.808505 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:12:20.808639 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:20.808640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:20.808661 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:20.808753 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:20.809327 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.810097 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.811436 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:20.811763 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:20.811904 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:20.816076 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:46575: BlockingRecv error: recv got EOF from 127.31.227.3:46575 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 00:12:21 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:21 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:21.483454 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:21.478490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:38721)
0812 00:12:21.478649 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:21.478662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:21.478756 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:21.478941 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:21.478943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:21.479219 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:21.479283 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:21.479286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.480012 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.480014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:21.480586 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:21.480588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.480987 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.480988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:21.481031 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.481600 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:21.481606 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:21.483387 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":118,"threads_started":1}
W20260812 00:12:21.483562 16249 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:21.478627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38721 (local address 127.31.227.62:46721)
0812 00:12:21.478756 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:21.478758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:21.478769 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:21.478821 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:21.478822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:21.478837 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:21.478946 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:21.479388 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.479918 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.480695 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.480907 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.480974 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.483486 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:38721: BlockingRecv error: recv got EOF from 127.31.227.1:38721 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
Aug 12 00:12:21 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:21 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:21.623847 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:21.619145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:53997)
0812 00:12:21.619337 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:21.619350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:21.619457 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:21.619635 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:21.619636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:21.619885 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:21.619964 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:21.619967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.620632 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.620632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:21.621161 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:21.621162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.621494 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.621495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:21.621562 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.622001 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:21.622006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:21.623788 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
W20260812 00:12:21.624017 16249 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:21.619297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:53997 (local address 127.31.227.62:46721)
0812 00:12:21.619373 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:21.619376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:21.619433 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:12:21.619534 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:21.619536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:21.619550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:21.619616 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:21.620048 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.620563 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.621233 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.621426 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.621476 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.623930 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:53997: BlockingRecv error: recv got EOF from 127.31.227.2:53997 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 00:12:21 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:21 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:21.821590 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:21.816754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:47881)
0812 00:12:21.816913 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:21.816933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:21.816991 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:21.817148 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:21.817149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:21.817327 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:21.817384 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:21.817386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.818158 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.818159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:21.818711 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:21.818713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.819059 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.819060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:21.819103 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.819623 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:21.819628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:21.821452 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
W20260812 00:12:21.821640 16249 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:21.816948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:47881 (local address 127.31.227.62:46721)
0812 00:12:21.817019 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:21.817021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:21.817033 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:12:21.817062 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:21.817063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:21.817075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:21.817126 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:21.817483 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.818071 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.818797 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.818984 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.819039 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.821573 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:47881: BlockingRecv error: recv got EOF from 127.31.227.3:47881 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 00:12:22 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:22 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:22.488646 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.483965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:42031)
0812 00:12:22.484117 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:22.484127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:22.484181 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:22.484314 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:22.484315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:22.484501 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:22.484554 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.484556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.485226 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.485226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.485785 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.485786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.486152 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.486153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.486200 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.486724 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:22.486729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:22.488584 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260812 00:12:22.488802 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:22.484047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:42031 (local address 127.31.227.62:46721)
0812 00:12:22.484155 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:22.484157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:22.484171 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:12:22.484245 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:22.484246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:22.484257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:22.484303 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:22.484616 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.485153 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.485859 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.486074 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.486148 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.488706 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:42031: BlockingRecv error: recv got EOF from 127.31.227.1:42031 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
Aug 12 00:12:22 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:22 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:22.628836 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.624347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:47891)
0812 00:12:22.624506 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:22.624517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:22.624577 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:22.624707 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:22.624708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:22.624888 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:22.624939 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.624941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.625618 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.625619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.626161 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.626163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.626526 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.626527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.626569 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.627013 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:22.627018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:22.628776 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
W20260812 00:12:22.628937 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:22.624416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:47891 (local address 127.31.227.62:46721)
0812 00:12:22.624489 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:22.624491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:22.624575 (+ 84us) server_negotiation.cc:407] Connection header received
0812 00:12:22.624624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:22.624626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:22.624637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:22.624689 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:22.625004 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.625552 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.626239 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.626462 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.626519 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.628870 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:47891: BlockingRecv error: recv got EOF from 127.31.227.2:47891 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 00:12:22 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:22 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:22.827075 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.822197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:36609)
0812 00:12:22.822381 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:22.822392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:22.822455 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:22.822600 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:22.822601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:22.822776 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:22.822830 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.822832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.823580 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.823581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.824112 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.824114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.824497 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.824499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.824563 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.825103 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:22.825109 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:22.826974 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
W20260812 00:12:22.827114 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:22.822289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:36609 (local address 127.31.227.62:46721)
0812 00:12:22.822376 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:22.822378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:22.822456 (+ 78us) server_negotiation.cc:407] Connection header received
0812 00:12:22.822520 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:22.822521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:22.822535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:22.822580 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:22.822911 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.823508 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.824191 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.824381 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.824460 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.827052 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:36609: BlockingRecv error: recv got EOF from 127.31.227.3:36609 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 00:12:23 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:23 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:23.494009 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.489105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:49477)
0812 00:12:23.489269 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:23.489280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:23.489334 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:23.489474 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:23.489475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:23.489671 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:23.489725 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.489726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.490356 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.490357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.490887 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.490889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.491244 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.491245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.491289 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.491770 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:23.491774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:23.493905 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
W20260812 00:12:23.494073 16265 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:23.489187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:49477 (local address 127.31.227.62:46721)
0812 00:12:23.489330 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.489332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.489341 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:23.489387 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.489387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.489399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.489459 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.489785 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.490290 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.490953 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.491143 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.491230 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.493999 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:49477: BlockingRecv error: recv got EOF from 127.31.227.1:49477 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
Aug 12 00:12:23 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:23 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:23.634399 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.629423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:34593)
0812 00:12:23.629588 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:23.629605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:23.629668 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:23.629851 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:23.629853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:23.630055 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:23.630107 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.630109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.630792 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.630793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.631340 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.631342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.631706 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.631707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.631774 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.632279 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:23.632289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:23.634299 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":156,"threads_started":1}
W20260812 00:12:23.634476 16265 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:23.629547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34593 (local address 127.31.227.62:46721)
0812 00:12:23.629632 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.629634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.629653 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:12:23.629749 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.629750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.629763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.629842 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.630190 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.630706 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.631430 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.631623 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.631697 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.634410 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:34593: BlockingRecv error: recv got EOF from 127.31.227.2:34593 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 00:12:23 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:23 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:23.832198 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.827593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:58187)
0812 00:12:23.827758 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:23.827770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:23.827832 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:23.827977 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:23.827978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:23.828149 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:23.828202 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.828203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.828825 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.828826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.829355 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.829356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.829736 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.829737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.829783 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.830294 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:23.830299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:23.832137 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
W20260812 00:12:23.832368 16265 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:23.827713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58187 (local address 127.31.227.62:46721)
0812 00:12:23.827775 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.827777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.827819 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:12:23.827892 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.827893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.827914 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.827958 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.828263 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.828756 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.829456 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.829667 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.829724 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.832293 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:58187: BlockingRecv error: recv got EOF from 127.31.227.3:58187 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 12 00:12:24 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:24 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:24.502245 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.494580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:44077)
0812 00:12:24.494725 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:24.494736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:24.494796 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:24.494926 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:24.494927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:24.495105 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:24.495159 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.495161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.495828 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.495829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.496360 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.496361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.496699 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.496700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.496744 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.500245 (+ 3501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:24.500250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:24.502180 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260812 00:12:24.502432 16273 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:24.494609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:44077 (local address 127.31.227.62:46721)
0812 00:12:24.494741 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.494743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.494779 (+ 36us) server_negotiation.cc:407] Connection header received
0812 00:12:24.494839 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.494840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.494853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.494915 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:24.495222 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.495750 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.496441 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.496633 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.496684 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.502313 (+ 5629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:44077: BlockingRecv error: recv got EOF from 127.31.227.1:44077 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
Aug 12 00:12:24 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:24 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:24.639742 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.635042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:37527)
0812 00:12:24.635213 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:24.635226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:24.635300 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:24.635460 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:24.635463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:24.635675 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:24.635730 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.635733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.636422 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.636424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.636959 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.636960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.637291 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.637292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.637337 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.637839 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:24.637845 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:24.639645 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
W20260812 00:12:24.639801 16273 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:24.635165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:37527 (local address 127.31.227.62:46721)
0812 00:12:24.635218 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.635220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.635279 (+ 59us) server_negotiation.cc:407] Connection header received
0812 00:12:24.635369 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.635369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.635382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.635436 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:24.635805 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.636297 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.637037 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.637225 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.637279 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.639738 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:37527: BlockingRecv error: recv got EOF from 127.31.227.2:37527 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 12 00:12:24 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:24 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:24.837610 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.832712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:37463)
0812 00:12:24.832855 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:24.832868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:24.832958 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:24.833128 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:24.833129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:24.833323 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:24.833376 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.833378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.834111 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.834112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.834643 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.834645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.835005 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.835006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.835068 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.835574 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:24.835579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:24.837487 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260812 00:12:24.837667 16273 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:24.832829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:37463 (local address 127.31.227.62:46721)
0812 00:12:24.832901 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.832904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.832937 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:12:24.833035 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.833036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.833057 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.833120 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:24.833461 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.834022 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.834726 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.834922 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.834994 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.837603 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:37463: BlockingRecv error: recv got EOF from 127.31.227.3:37463 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 00:12:25 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:25 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:25.507920 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.502845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:57059)
0812 00:12:25.503003 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.503018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:25.503074 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:25.503221 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:25.503222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:25.503402 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:25.503454 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.503456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.504098 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.504099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.504625 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.504626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.505058 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.505060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.505118 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.505723 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:25.505729 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:25.507815 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
W20260812 00:12:25.507979 16281 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:25.502922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:57059 (local address 127.31.227.62:46721)
0812 00:12:25.503076 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:25.503078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:25.503117 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:12:25.503140 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:25.503140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:25.503151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:25.503205 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:25.503525 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.504029 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.504718 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.504963 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.505028 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.507905 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:57059: BlockingRecv error: recv got EOF from 127.31.227.1:57059 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
Aug 12 00:12:25 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:25 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:25.645445 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.640351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:37947)
0812 00:12:25.640502 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.640516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:25.640591 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:25.640774 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:25.640775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:25.640968 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:25.641021 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.641023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.641813 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.641815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.642386 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.642387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.642769 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.642770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.642823 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.643395 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:25.643401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:25.645348 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
W20260812 00:12:25.645537 16281 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:25.640470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:37947 (local address 127.31.227.62:46721)
0812 00:12:25.640532 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:25.640534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:25.640573 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:12:25.640655 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:25.640656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:25.640670 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:25.640726 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:25.641122 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.641737 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.642469 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.642668 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.642737 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.645448 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:37947: BlockingRecv error: recv got EOF from 127.31.227.2:37947 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 12 00:12:25 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:25 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:25.847288 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.838151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:42701)
0812 00:12:25.838347 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.838358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:25.838420 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:25.838571 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:25.838572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:25.838754 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:25.838809 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.838811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.839467 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.839469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.840008 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.840009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.840372 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.840374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.840427 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.845101 (+ 4674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:25.845107 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:25.847225 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
W20260812 00:12:25.847388 16281 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:25.838239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:42701 (local address 127.31.227.62:46721)
0812 00:12:25.838292 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:25.838294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:25.838405 (+ 111us) server_negotiation.cc:407] Connection header received
0812 00:12:25.838483 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:25.838483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:25.838498 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:25.838543 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:25.838881 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.839381 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.840088 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.840277 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.840333 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.847338 (+ 7005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:42701: BlockingRecv error: recv got EOF from 127.31.227.3:42701 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 00:12:26 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:26 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:26.513275 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:26.508477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:34935)
0812 00:12:26.508647 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:26.508657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:26.508744 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:26.508865 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:26.508866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:26.509076 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:26.509131 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:26.509133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.509786 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.509787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:26.510446 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:26.510449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.510786 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.510788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:26.510838 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:26.511417 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:26.511422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:26.513216 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
W20260812 00:12:26.513408 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:26.508554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:34935 (local address 127.31.227.62:46721)
0812 00:12:26.508677 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:26.508679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:26.508722 (+ 43us) server_negotiation.cc:407] Connection header received
0812 00:12:26.508790 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:26.508790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:26.508803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:26.508862 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:26.509182 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.509710 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.510512 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.510711 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.510767 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:26.513316 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:34935: BlockingRecv error: recv got EOF from 127.31.227.1:34935 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
Aug 12 00:12:26 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:26 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:26.651041 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:26.646038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:38923)
0812 00:12:26.646166 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:26.646182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:26.646244 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:26.646384 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:26.646385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:26.646549 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:26.646603 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:26.646605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.647250 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.647251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:26.647774 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:26.647775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.648102 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.648103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:26.648143 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:26.648643 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:26.648649 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:26.650928 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
W20260812 00:12:26.651072 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:26.646139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:38923 (local address 127.31.227.62:46721)
0812 00:12:26.646200 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:26.646202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:26.646232 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:12:26.646294 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:26.646294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:26.646306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:26.646365 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:26.646670 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.647186 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.647844 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.648040 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.648091 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:26.651006 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:38923: BlockingRecv error: recv got EOF from 127.31.227.2:38923 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 12 00:12:26 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:26 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:26.852424 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:26.848016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:39825)
0812 00:12:26.848147 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:26.848159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:26.848228 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:26.848351 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:26.848352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:26.848518 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:26.848569 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:26.848571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.849189 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.849190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:26.849759 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:26.849761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.850120 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.850120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:26.850161 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:26.850597 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:26.850602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:26.852360 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
W20260812 00:12:26.852551 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:26.848090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:39825 (local address 127.31.227.62:46721)
0812 00:12:26.848169 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:26.848171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:26.848232 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:12:26.848273 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:26.848274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:26.848289 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:26.848344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:26.848623 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.849121 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.849843 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:26.850049 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:26.850112 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:26.852453 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:39825: BlockingRecv error: recv got EOF from 127.31.227.3:39825 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 00:12:27 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:27 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:27.518522 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.513822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:42521)
0812 00:12:27.513986 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.513997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.514055 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.514226 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.514227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.514424 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.514478 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.514480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.515196 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.515197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.515731 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.515733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.516092 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.516094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.516147 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.516642 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.516647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.518461 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
W20260812 00:12:27.518666 16293 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:27.513979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:42521 (local address 127.31.227.62:46721)
0812 00:12:27.514100 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.514101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.514110 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:27.514136 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.514136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.514148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.514208 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.514594 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.515117 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.515800 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.515999 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.516053 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.518572 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:42521: BlockingRecv error: recv got EOF from 127.31.227.1:42521 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
Aug 12 00:12:27 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:27 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:27.656536 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.651543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:60877)
0812 00:12:27.651726 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.651740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.651830 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.652021 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.652022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.652215 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.652267 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.652269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.653099 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.653100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.653711 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.653714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.654095 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.654096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.654158 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.654594 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.654599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.656436 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
W20260812 00:12:27.656585 16293 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:27.651650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60877 (local address 127.31.227.62:46721)
0812 00:12:27.651730 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.651733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.651815 (+ 82us) server_negotiation.cc:407] Connection header received
0812 00:12:27.651927 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.651928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.651944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.652011 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.652345 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.653012 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.653791 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.654008 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.654092 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.656517 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:60877: BlockingRecv error: recv got EOF from 127.31.227.2:60877 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 00:12:27 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:27 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:27.857903 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.852915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:42359)
0812 00:12:27.853070 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.853083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.853156 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.853345 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.853346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.853589 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.853651 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.853654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.854357 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.854359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.854906 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.854907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.855328 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.855329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.855392 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.855909 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.855915 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.857797 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20260812 00:12:27.857944 16293 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:27.852994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:42359 (local address 127.31.227.62:46721)
0812 00:12:27.853074 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.853076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.853215 (+ 139us) server_negotiation.cc:407] Connection header received
0812 00:12:27.853249 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.853249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.853264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.853321 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.853727 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.854265 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.854990 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.855265 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.855328 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.857881 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:42359: BlockingRecv error: recv got EOF from 127.31.227.3:42359 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 00:12:28 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:28 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:28.523423 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.518980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:32839)
0812 00:12:28.519118 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.519127 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.519171 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.519331 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.519332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.519516 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.519566 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.519567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.520201 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.520202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.520741 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.520742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.521068 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.521069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.521115 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.521583 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.521588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.523347 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
W20260812 00:12:28.523528 16298 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:28.519069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:32839 (local address 127.31.227.62:46721)
0812 00:12:28.519200 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.519202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.519211 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:28.519233 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.519234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.519245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.519313 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.519631 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.520136 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.520808 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.520996 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.521045 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.523453 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:32839: BlockingRecv error: recv got EOF from 127.31.227.1:32839 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Aug 12 00:12:28 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:28 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:28.661384 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.656999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:38587)
0812 00:12:28.657141 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.657151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.657211 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.657350 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.657351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.657548 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.657603 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.657605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.658230 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.658231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.658747 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.658748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.659084 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.659085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.659126 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.659590 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.659595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.661316 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
W20260812 00:12:28.661477 16298 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:28.657092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:38587 (local address 127.31.227.62:46721)
0812 00:12:28.657150 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.657152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.657209 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:12:28.657257 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.657258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.657270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.657334 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.657665 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.658160 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.658821 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.659021 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.659076 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.661412 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:38587: BlockingRecv error: recv got EOF from 127.31.227.2:38587 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 00:12:28 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:28 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:28.863664 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.858471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:34125)
0812 00:12:28.858625 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.858636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.858695 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.858835 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.858836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.859020 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.859068 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.859071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.859888 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.859889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.860749 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.860752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.861160 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.861161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.861204 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.861667 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.861672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.863548 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260812 00:12:28.863694 16298 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:28.858547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:34125 (local address 127.31.227.62:46721)
0812 00:12:28.858616 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.858618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.858696 (+ 78us) server_negotiation.cc:407] Connection header received
0812 00:12:28.858742 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.858743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.858754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.858818 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.859133 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.859791 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.860831 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.861093 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.861149 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.863627 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:34125: BlockingRecv error: recv got EOF from 127.31.227.3:34125 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 00:12:29 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:29 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:29.532696 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:29.523898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:56915)
0812 00:12:29.524086 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:29.524097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:29.524162 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:29.524310 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:29.524311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:29.524491 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:29.524544 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:29.524546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.525173 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.525174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:29.525774 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:29.525777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.526103 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.526104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:29.526151 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:29.530567 (+ 4416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:29.530573 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:29.532631 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
W20260812 00:12:29.532832 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:29.523992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:56915 (local address 127.31.227.62:46721)
0812 00:12:29.524106 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:29.524108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:29.524157 (+ 49us) server_negotiation.cc:407] Connection header received
0812 00:12:29.524225 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:29.524225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:29.524235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:29.524294 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:29.524608 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.525108 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.525843 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.526039 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.526089 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:29.532737 (+ 6648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:56915: BlockingRecv error: recv got EOF from 127.31.227.1:56915 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
Aug 12 00:12:29 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:29 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:29.666792 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:29.662012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:36643)
0812 00:12:29.662190 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:29.662199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:29.662275 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:29.662460 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:29.662462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:29.662656 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:29.662710 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:29.662711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.663380 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.663381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:29.663915 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:29.663917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.664276 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.664277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:29.664361 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:29.664808 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:29.664813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:29.666686 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
W20260812 00:12:29.666847 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:29.662058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:36643 (local address 127.31.227.62:46721)
0812 00:12:29.662144 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:29.662147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:29.662260 (+ 113us) server_negotiation.cc:407] Connection header received
0812 00:12:29.662366 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:29.662367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:29.662386 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:29.662450 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:29.662790 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.663297 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.664003 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.664201 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.664276 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:29.666773 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:36643: BlockingRecv error: recv got EOF from 127.31.227.2:36643 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 00:12:29 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:29 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:29.868851 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:29.864174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:34929)
0812 00:12:29.864332 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:29.864348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:29.864410 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:29.864545 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:29.864546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:29.864710 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:29.864762 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:29.864763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.865383 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.865384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:29.866009 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:29.866012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.866389 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.866390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:29.866447 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:29.866909 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:29.866913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:29.868755 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
W20260812 00:12:29.868921 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:29.864281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:34929 (local address 127.31.227.62:46721)
0812 00:12:29.864338 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:29.864340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:29.864394 (+ 54us) server_negotiation.cc:407] Connection header received
0812 00:12:29.864458 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:29.864459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:29.864472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:29.864530 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:29.864825 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.865318 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.866112 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:29.866302 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:29.866392 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:29.868854 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:34929: BlockingRecv error: recv got EOF from 127.31.227.3:34929 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 12 00:12:30 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:30 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:30.538107 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:30.533279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:57011)
0812 00:12:30.533430 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:30.533441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:30.533544 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:30.533712 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:30.533713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:30.533936 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:30.533990 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:30.533992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.534720 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.534720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:30.535257 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:30.535259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.535586 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.535587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:30.535627 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.536075 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:30.536079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:30.538009 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
W20260812 00:12:30.538378 16311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:30.533368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:57011 (local address 127.31.227.62:46721)
0812 00:12:30.533495 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:30.533497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:30.533506 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:30.533612 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:30.533613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:30.533627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:30.533695 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:30.534063 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.534651 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.535320 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.535524 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.535574 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.538134 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:57011: BlockingRecv error: recv got EOF from 127.31.227.1:57011 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Aug 12 00:12:30 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:30 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:30.672703 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:30.667308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:39541)
0812 00:12:30.667485 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:30.667497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:30.667593 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:30.667751 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:30.667752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:30.667947 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:30.668002 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:30.668004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.668797 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.668799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:30.669454 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:30.669457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.669911 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.669912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:30.669960 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.670456 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:30.670462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:30.672633 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
W20260812 00:12:30.672823 16311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:30.667365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:39541 (local address 127.31.227.62:46721)
0812 00:12:30.667451 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:30.667453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:30.667571 (+ 118us) server_negotiation.cc:407] Connection header received
0812 00:12:30.667670 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:30.667671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:30.667684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:30.667743 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:30.668088 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.668704 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.669566 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.669838 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.669893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.672726 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:39541: BlockingRecv error: recv got EOF from 127.31.227.2:39541 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 00:12:30 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:30 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:30.874217 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:30.869382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:57869)
0812 00:12:30.869589 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:30.869607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:30.869671 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:30.869793 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:30.869793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:30.869972 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:30.870023 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:30.870025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.870666 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.870667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:30.871221 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:30.871222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.871612 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.871613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:30.871676 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.872206 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:30.872211 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:30.874147 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":117,"threads_started":1}
W20260812 00:12:30.874377 16311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:30.869442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:57869 (local address 127.31.227.62:46721)
0812 00:12:30.869538 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:30.869540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:30.869667 (+ 127us) server_negotiation.cc:407] Connection header received
0812 00:12:30.869717 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:30.869718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:30.869733 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:30.869786 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:30.870077 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.870590 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.871306 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.871519 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.871575 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.874291 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:57869: BlockingRecv error: recv got EOF from 127.31.227.3:57869 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 00:12:31 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:31 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:31.543474 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:31.538724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:58247)
0812 00:12:31.538887 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:31.538900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:31.538970 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:31.539140 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:31.539142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:31.539322 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:31.539400 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:31.539402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.540059 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.540059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:31.540634 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:31.540635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.540956 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.540957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:31.540999 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:31.541476 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:31.541481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:31.543414 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
W20260812 00:12:31.543586 16316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:31.538829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:58247 (local address 127.31.227.62:46721)
0812 00:12:31.538972 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:31.538974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:31.539002 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:12:31.539025 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:31.539026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:31.539036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:31.539088 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:31.539470 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.539993 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.540701 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.540895 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.540943 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:31.543508 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:58247: BlockingRecv error: recv got EOF from 127.31.227.1:58247 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
Aug 12 00:12:31 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:31 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:31.678115 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:31.673372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:38219)
0812 00:12:31.673583 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:31.673597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:31.673672 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:31.673832 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:31.673833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:31.674072 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:31.674122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:31.674124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.674786 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.674787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:31.675328 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:31.675330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.675672 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.675673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:31.675717 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:31.676230 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:31.676235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:31.678043 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":125,"threads_started":1}
W20260812 00:12:31.678237 16316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:31.673452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:38219 (local address 127.31.227.62:46721)
0812 00:12:31.673553 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:31.673556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:31.673667 (+ 111us) server_negotiation.cc:407] Connection header received
0812 00:12:31.673740 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:31.673742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:31.673758 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:31.673815 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:31.674187 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.674717 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.675402 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.675604 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.675664 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:31.678155 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:38219: BlockingRecv error: recv got EOF from 127.31.227.2:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 00:12:31 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:31 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:31.879663 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:31.874715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:36433)
0812 00:12:31.874928 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:31.874938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:31.875020 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:31.875175 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:31.875176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:31.875370 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:31.875421 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:31.875423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.876097 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.876098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:31.876630 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:31.876632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.876997 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.876998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:31.877061 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:31.877692 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:31.877697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:31.879561 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
W20260812 00:12:31.879720 16316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:31.874781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:36433 (local address 127.31.227.62:46721)
0812 00:12:31.874875 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:31.874879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:31.874999 (+ 120us) server_negotiation.cc:407] Connection header received
0812 00:12:31.875091 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:31.875092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:31.875114 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:31.875168 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:31.875496 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.876005 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.876711 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:31.876911 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:31.876994 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:31.879651 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:36433: BlockingRecv error: recv got EOF from 127.31.227.3:36433 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 00:12:32 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:32 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:32.549221 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.544045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:49197)
0812 00:12:32.544177 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.544204 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.544268 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.544602 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.544603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.544790 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:32.544842 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.544844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.545602 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.545604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.546200 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.546203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.546563 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.546564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.546606 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.547122 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:32.547127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:32.549159 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
W20260812 00:12:32.549419 16323 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:32.544082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:49197 (local address 127.31.227.62:46721)
0812 00:12:32.544213 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.544215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.544250 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:12:32.544329 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.544330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.544343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.544591 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.544913 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.545482 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.546284 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.546496 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.546553 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.549311 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:49197: BlockingRecv error: recv got EOF from 127.31.227.1:49197 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Aug 12 00:12:32 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:32 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:32.683338 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.678661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:51247)
0812 00:12:32.678827 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.678839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.678927 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.679087 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.679088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.679279 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:32.679334 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.679336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.679972 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.679973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.680501 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.680502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.680814 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.680815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.680855 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.681355 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:32.681360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:32.683280 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
W20260812 00:12:32.683493 16323 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:32.678791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:51247 (local address 127.31.227.62:46721)
0812 00:12:32.678840 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.678843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.678921 (+ 78us) server_negotiation.cc:407] Connection header received
0812 00:12:32.679005 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.679006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.679020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.679081 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.679396 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.679904 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.680569 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.680752 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.680803 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.683399 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:51247: BlockingRecv error: recv got EOF from 127.31.227.2:51247 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 00:12:32 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:32 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:32.885120 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.880278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:35331)
0812 00:12:32.880449 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.880461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.880531 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.880674 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.880675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.880854 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:32.880907 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.880909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.881616 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.881617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.882162 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.882164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.882553 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.882554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.882606 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.883065 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:32.883070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:32.885022 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
W20260812 00:12:32.885308 16323 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:32.880358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:35331 (local address 127.31.227.62:46721)
0812 00:12:32.880430 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.880447 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.880531 (+ 84us) server_negotiation.cc:407] Connection header received
0812 00:12:32.880581 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.880582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.880596 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.880654 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.880978 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.881478 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.882236 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.882446 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.882500 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.885135 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:35331: BlockingRecv error: recv got EOF from 127.31.227.3:35331 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 00:12:33 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:33 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:33.554651 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.549738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:49495)
0812 00:12:33.549891 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.549903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.549976 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.550181 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.550182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.550387 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:33.550438 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.550440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.551149 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.551150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.551715 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.551717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.552095 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.552097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.552169 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.552634 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:33.552639 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:33.554537 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20260812 00:12:33.554698 16333 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:33.549786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:49495 (local address 127.31.227.62:46721)
0812 00:12:33.549910 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.549912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.549955 (+ 43us) server_negotiation.cc:407] Connection header received
0812 00:12:33.550064 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.550065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.550082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.550163 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.550555 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.551063 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.551806 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.552016 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.552095 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.554624 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:49495: BlockingRecv error: recv got EOF from 127.31.227.1:49495 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
Aug 12 00:12:33 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:33 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:33.688313 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.683814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:53833)
0812 00:12:33.684014 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.684032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.684101 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.684239 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.684240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.684449 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:33.684501 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.684503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.685135 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.685136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.685719 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.685721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.686042 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.686043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.686084 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.686562 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:33.686567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:33.688244 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
W20260812 00:12:33.688385 16333 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:33.683867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:53833 (local address 127.31.227.62:46721)
0812 00:12:33.683940 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.683942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.684098 (+ 156us) server_negotiation.cc:407] Connection header received
0812 00:12:33.684151 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.684152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.684164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.684215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.684565 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.685063 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.685789 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.685973 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.686029 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.688337 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:53833: BlockingRecv error: recv got EOF from 127.31.227.2:53833 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 00:12:33 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:33 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.3@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:33.890367 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.885684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.3:54371)
0812 00:12:33.885851 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.885862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.885936 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.886068 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.886069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.886254 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:33.886304 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.886305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.886929 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.886929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.887488 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.887490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.887804 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.887805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.887847 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.888289 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:33.888294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:33.890296 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
W20260812 00:12:33.890467 16333 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:33.885798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:54371 (local address 127.31.227.62:46721)
0812 00:12:33.885863 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.885865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.885933 (+ 68us) server_negotiation.cc:407] Connection header received
0812 00:12:33.885984 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.885990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.886003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.886048 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.886364 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.886859 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.887557 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.887738 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.887787 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.890398 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.3:54371: BlockingRecv error: recv got EOF from 127.31.227.3:54371 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 00:12:34 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:34 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.1@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:34.563489 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.555254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.1:56457)
0812 00:12:34.555421 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.555435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:34.555508 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:34.555669 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:34.555670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:34.555853 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:34.555907 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:34.555909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.556631 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.556632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:34.557198 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:34.557200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.557591 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.557592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:34.557639 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:34.561498 (+ 3859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:34.561503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:34.563427 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
W20260812 00:12:34.563618 16337 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:34.555364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:56457 (local address 127.31.227.62:46721)
0812 00:12:34.555538 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:34.555540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:34.555548 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:12:34.555573 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:34.555573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:34.555585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:34.555648 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:34.555979 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.556509 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.557288 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.557504 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.557585 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:34.563534 (+ 5949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.1:56457: BlockingRecv error: recv got EOF from 127.31.227.1:56457 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
Aug 12 00:12:34 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:12:34 dist-test-slave-zr1t krb5kdc[14521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.227.2@KRBTEST.COM for oryx/127.31.227.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:12:34.693878 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.689001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46721 (local address 127.31.227.2:56949)
0812 00:12:34.689200 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.689219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:34.689290 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:34.689462 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:34.689463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:34.689724 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:34.689796 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:34.689800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.690575 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.690576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:34.691133 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:34.691135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.691463 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.691463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:34.691506 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:34.691948 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:34.691953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:34.693801 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.227.62:46721: Server oryx/127.31.227.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":132,"threads_started":1}
W20260812 00:12:34.693990 16337 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:34.689121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:56949 (local address 127.31.227.62:46721)
0812 00:12:34.689208 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:34.689210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:34.689275 (+ 65us) server_negotiation.cc:407] Connection header received
0812 00:12:34.689358 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:34.689359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:34.689380 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:34.689426 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:34.689929 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.690482 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.691211 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.691396 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.691446 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:34.693899 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.227.2:56949: BlockingRecv error: recv got EOF from 127.31.227.2:56949 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 00:12:34.788374 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 15515
I20260812 00:12:34.802171 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 15758
I20260812 00:12:34.815102 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 16011
I20260812 00:12:34.823074 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 15406
2026-08-12T00:12:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17859 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T00:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:34Z Disabled control of system clock
I20260812 00:12:34.862916 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34903
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 00:12:34.946524 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:34.946727 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:34.946748 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:34.947955 16350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:34.947997 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:34.948010 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:34.948022 16350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:34.948035 16350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:34.949306 16350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:34.949510 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:34.949833 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:34.954777 16358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.954744 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.954900 16350 server_base.cc:1034] running on GCE node
W20260812 00:12:34.954766 16360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.955266 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:34.955564 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:34.956703 16350 hybrid_clock.cc:648] HybridClock initialized: now 1786493554956683 us; error 36 us; skew 500 ppm
I20260812 00:12:34.958093 16350 webserver.cc:460] Webserver started at http://127.31.227.62:38835/ using document root <none> and password file <none>
I20260812 00:12:34.958309 16350 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:34.958352 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:34.958446 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:34.959476 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "0b4c29549a9643e0a0031e8163aa7eeb"
format_stamp: "Formatted at 2026-08-12 00:12:34 on dist-test-slave-zr1t"
server_key: "abcd47ba0d059e13b828be3f410eb401"
server_key_iv: "4fd7617c3130bb1cf25a595de19a664e"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.959789 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "0b4c29549a9643e0a0031e8163aa7eeb"
format_stamp: "Formatted at 2026-08-12 00:12:34 on dist-test-slave-zr1t"
server_key: "abcd47ba0d059e13b828be3f410eb401"
server_key_iv: "4fd7617c3130bb1cf25a595de19a664e"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.961148 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:12:34.961978 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.962167 16350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:34.962248 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "0b4c29549a9643e0a0031e8163aa7eeb"
format_stamp: "Formatted at 2026-08-12 00:12:34 on dist-test-slave-zr1t"
server_key: "abcd47ba0d059e13b828be3f410eb401"
server_key_iv: "4fd7617c3130bb1cf25a595de19a664e"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.962318 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:34.974017 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:34.974303 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:34.974407 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:34.978394 16350 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34903
I20260812 00:12:34.978427 16455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34903 every 8 connection(s)
I20260812 00:12:34.978796 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:34.979578 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:34.982455 16457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb: Bootstrap starting.
I20260812 00:12:34.983091 16457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:34.983420 16457 log.cc:824] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:34.984136 16457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb: No bootstrap required, opened a new log
I20260812 00:12:34.985848 16457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34903 } }
I20260812 00:12:34.985991 16457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:34.986013 16457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4c29549a9643e0a0031e8163aa7eeb, State: Initialized, Role: FOLLOWER
I20260812 00:12:34.986169 16457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34903 } }
I20260812 00:12:34.986245 16457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:34.986286 16457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:34.986346 16457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:34.986907 16457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34903 } }
I20260812 00:12:34.987026 16457 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4c29549a9643e0a0031e8163aa7eeb; no voters:
I20260812 00:12:34.987213 16457 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:34.987331 16462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:34.987588 16462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [term 1 LEADER]: Becoming Leader. State: Replica: 0b4c29549a9643e0a0031e8163aa7eeb, State: Running, Role: LEADER
I20260812 00:12:34.987665 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 16350
I20260812 00:12:34.987699 16462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34903 } }
I20260812 00:12:34.987778 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:34.988003 32652 external_mini_cluster.cc:1428] Setting key 81e76d90272fb439920294156b249e2b
I20260812 00:12:34.988138 16457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:34.988384 16463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34903 } } }
I20260812 00:12:34.988505 16463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:34.988106 16464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4c29549a9643e0a0031e8163aa7eeb. Latest consensus state: current_term: 1 leader_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c29549a9643e0a0031e8163aa7eeb" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34903 } } }
I20260812 00:12:34.988736 16464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:34.989027 16469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:34.989609 16469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:34.991436 16469 catalog_manager.cc:1347] Generated new cluster ID: 8cc1cc86c80648fdabd411a220ecbab8
I20260812 00:12:34.991493 16469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:34.995648 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.988700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.31.227.62:34903)
0812 00:12:34.988914 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:34.988924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:34.990031 (+ 1107us) server_negotiation.cc:407] Connection header received
0812 00:12:34.990356 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:34.990365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:34.990461 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:34.990610 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:34.990802 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.992164 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.993326 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:34.993986 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:34.994101 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:34.994922 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:34.994927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:34.994927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:34.995145 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:34.995188 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:34.995202 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:34.995269 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:34.995354 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 00:12:34.995438 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260812 00:12:35.000464 16469 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:35.001173 16469 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:35.006060 16469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4c29549a9643e0a0031e8163aa7eeb: Generated new TSK 0
I20260812 00:12:35.006263 16469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:35.010303 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34903
--builtin_ntp_servers=127.31.227.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 00:12:35.101087 16496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.101327 16496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.101365 16496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:35.102566 16496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:35.102630 16496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:35.102672 16496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:35.104018 16496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:35.104287 16496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:35.104599 16496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:35.109740 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.109818 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.110054 16496 server_base.cc:1034] running on GCE node
W20260812 00:12:35.109849 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.110342 16496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:35.110621 16496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:35.111786 16496 hybrid_clock.cc:648] HybridClock initialized: now 1786493555111774 us; error 29 us; skew 500 ppm
I20260812 00:12:35.113225 16496 webserver.cc:460] Webserver started at http://127.31.227.1:45139/ using document root <none> and password file <none>
I20260812 00:12:35.113477 16496 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:35.113559 16496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:35.113686 16496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:35.115135 16496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "eab96c31d13644b0b101f337736384e3"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "afeae511f2b7a76ad294f2b884845c48"
server_key_iv: "36f0178eabae1e8efbf449619a467221"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.115581 16496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "eab96c31d13644b0b101f337736384e3"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "afeae511f2b7a76ad294f2b884845c48"
server_key_iv: "36f0178eabae1e8efbf449619a467221"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.117430 16496 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 00:12:35.118374 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.118605 16496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:35.118717 16496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "eab96c31d13644b0b101f337736384e3"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "afeae511f2b7a76ad294f2b884845c48"
server_key_iv: "36f0178eabae1e8efbf449619a467221"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.118798 16496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:35.147789 16496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:35.148106 16496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:35.148218 16496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:35.148437 16496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:35.148716 16496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:35.148746 16496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.148768 16496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:35.148780 16496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.155344 16496 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:43671
I20260812 00:12:35.155405 16746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:43671 every 8 connection(s)
I20260812 00:12:35.155743 16496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:35.157724 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 16496
I20260812 00:12:35.157804 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:35.158010 32652 external_mini_cluster.cc:1428] Setting key 85c0cf3bd89d8d40f8bed892aeae7662
I20260812 00:12:35.159045 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34903
--builtin_ntp_servers=127.31.227.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 00:12:35.161893 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.156494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:58399 (local address 127.31.227.62:34903)
0812 00:12:35.156557 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.156559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.156789 (+ 230us) server_negotiation.cc:407] Connection header received
0812 00:12:35.157364 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.157366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.157382 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.157431 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.158329 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.159026 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.160100 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.160457 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.160543 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.161605 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.161606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.161606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:35.161613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.161631 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:35.161636 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.161694 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.161804 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.161829 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:12:35.162290 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.156490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34903 (local address 127.31.227.1:58399)
0812 00:12:35.156669 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.156703 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.157254 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.157560 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.157565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.157575 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.158223 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.158230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.159148 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.159150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.160009 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.160011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.160569 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.160571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.160762 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.161408 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:35.161409 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.161450 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:35.161482 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.161712 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.161719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.161723 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.161795 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.161923 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260812 00:12:35.162875 16749 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34903
I20260812 00:12:35.162992 16749 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:35.163203 16749 heartbeater.cc:511] Master 127.31.227.62:34903 requested a full tablet report, sending...
I20260812 00:12:35.163730 16394 ts_manager.cc:194] Registered new tserver with Master: eab96c31d13644b0b101f337736384e3 (127.31.227.1:43671)
I20260812 00:12:35.164414 16394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.1:58399
W20260812 00:12:35.244692 16753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.244900 16753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.244923 16753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:35.246037 16753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:35.246074 16753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:35.246099 16753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:35.247378 16753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:35.247581 16753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:35.247849 16753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:35.253190 16764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.253162 16766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.253190 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.253486 16753 server_base.cc:1034] running on GCE node
I20260812 00:12:35.253772 16753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:35.254043 16753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:35.255245 16753 hybrid_clock.cc:648] HybridClock initialized: now 1786493555255236 us; error 37 us; skew 500 ppm
I20260812 00:12:35.256620 16753 webserver.cc:460] Webserver started at http://127.31.227.2:46365/ using document root <none> and password file <none>
I20260812 00:12:35.256906 16753 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:35.256980 16753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:35.257144 16753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:35.258332 16753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "d606a934f0494a3dac6073a55837403b"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "6f59df18d767349902fcc04f2b4f28cb"
server_key_iv: "19fae6cf5cbae60fe33cb7458dc967e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.258713 16753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "d606a934f0494a3dac6073a55837403b"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "6f59df18d767349902fcc04f2b4f28cb"
server_key_iv: "19fae6cf5cbae60fe33cb7458dc967e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.260159 16753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:12:35.261027 16787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.261245 16753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:35.261361 16753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "d606a934f0494a3dac6073a55837403b"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "6f59df18d767349902fcc04f2b4f28cb"
server_key_iv: "19fae6cf5cbae60fe33cb7458dc967e1"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.261442 16753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:35.276846 16753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:35.277180 16753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:35.277330 16753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:35.277628 16753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:35.277987 16753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:35.278043 16753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.278084 16753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:35.278112 16753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.283705 16753 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:41803
I20260812 00:12:35.283828 16987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:41803 every 8 connection(s)
I20260812 00:12:35.284133 16753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:35.285768 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 16753
I20260812 00:12:35.285877 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:35.286105 32652 external_mini_cluster.cc:1428] Setting key 4573f532fd4d1eb328d6ea65016502e1
I20260812 00:12:35.287168 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34903
--builtin_ntp_servers=127.31.227.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 00:12:35.290218 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.284873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46183 (local address 127.31.227.62:34903)
0812 00:12:35.284947 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.284950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.285255 (+ 305us) server_negotiation.cc:407] Connection header received
0812 00:12:35.285869 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.285871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.285884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.285957 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.286847 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.287618 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.288585 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.288920 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.289001 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.289924 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.289925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.289926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:35.289931 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.289948 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:35.289954 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.290017 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.290134 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.290159 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:12:35.290508 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.284870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34903 (local address 127.31.227.2:46183)
0812 00:12:35.285096 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.285157 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.285751 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.286047 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.286051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.286060 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.286736 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.286744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.287705 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.287706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.288498 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.288500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.289011 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.289012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.289166 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.289788 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:35.289789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.289818 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:35.289840 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.290049 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.290056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.290060 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.290134 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.290244 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260812 00:12:35.291010 16989 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34903
I20260812 00:12:35.291106 16989 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:35.291303 16989 heartbeater.cc:511] Master 127.31.227.62:34903 requested a full tablet report, sending...
I20260812 00:12:35.291697 16394 ts_manager.cc:194] Registered new tserver with Master: d606a934f0494a3dac6073a55837403b (127.31.227.2:41803)
I20260812 00:12:35.292085 16394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.2:46183
W20260812 00:12:35.371954 16993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.372205 16993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.372241 16993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:35.373404 16993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:35.373468 16993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:35.373508 16993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:35.374867 16993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:35.375138 16993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:35.375453 16993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:35.380554 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.380581 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.380658 16993 server_base.cc:1034] running on GCE node
W20260812 00:12:35.380561 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.381021 16993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:35.381263 16993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:35.382434 16993 hybrid_clock.cc:648] HybridClock initialized: now 1786493555382414 us; error 37 us; skew 500 ppm
I20260812 00:12:35.383819 16993 webserver.cc:460] Webserver started at http://127.31.227.3:40235/ using document root <none> and password file <none>
I20260812 00:12:35.384034 16993 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:35.384078 16993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:35.384176 16993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:35.385281 16993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "464cefefd1b64bdfb5b418b26414ff7e"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "60c23868e1ef93ec92c96bb50f400acc"
server_key_iv: "b337884439ca10074dac96f709b33c77"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.385695 16993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "464cefefd1b64bdfb5b418b26414ff7e"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "60c23868e1ef93ec92c96bb50f400acc"
server_key_iv: "b337884439ca10074dac96f709b33c77"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.386951 16993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:35.387604 17025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.387775 16993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:35.387866 16993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "464cefefd1b64bdfb5b418b26414ff7e"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "60c23868e1ef93ec92c96bb50f400acc"
server_key_iv: "b337884439ca10074dac96f709b33c77"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.387938 16993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:35.402959 16993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:35.403319 16993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:35.403479 16993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:35.403750 16993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:35.404134 16993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:35.404170 16993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.404194 16993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:35.404243 16993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.410449 16993 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:44491
I20260812 00:12:35.410532 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:44491 every 8 connection(s)
I20260812 00:12:35.410909 16993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:35.413630 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 16993
I20260812 00:12:35.413744 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:35.413952 32652 external_mini_cluster.cc:1428] Setting key 4ae81242cbc5b9c6b8e3419f256a20e6
I20260812 00:12:35.416872 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.411790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:45083 (local address 127.31.227.62:34903)
0812 00:12:35.411877 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.411879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.412107 (+ 228us) server_negotiation.cc:407] Connection header received
0812 00:12:35.412604 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.412606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.412617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.412690 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.413435 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.414236 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.415570 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.415756 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.415813 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.416631 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.416632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.416633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:35.416637 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.416650 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:35.416654 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.416698 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.416785 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.416811 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 00:12:35.417148 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.411709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34903 (local address 127.31.227.3:45083)
0812 00:12:35.411943 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.411987 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.412500 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.412755 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.412759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.412767 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.413297 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.413302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.414399 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.414400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.415494 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.415497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.415837 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.415838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.415961 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.416492 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:35.416493 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.416523 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:35.416548 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.416713 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.416718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.416720 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.416775 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.416863 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260812 00:12:35.417850 17230 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34903
I20260812 00:12:35.417965 17230 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:35.418181 17230 heartbeater.cc:511] Master 127.31.227.62:34903 requested a full tablet report, sending...
I20260812 00:12:35.418584 16394 ts_manager.cc:194] Registered new tserver with Master: 464cefefd1b64bdfb5b418b26414ff7e (127.31.227.3:44491)
I20260812 00:12:35.418921 16394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.3:45083
I20260812 00:12:35.425944 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:12:35.427815 16478 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:35.427329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.31.227.62:34903)
0812 00:12:35.427441 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.427443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.427466 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:12:35.427513 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.427513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.427526 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.427566 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.427728 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40970: BlockingRecv error: recv got EOF from 127.0.0.1:40970 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 00:12:35.427829 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 16496
I20260812 00:12:35.440186 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 16753
I20260812 00:12:35.457607 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 16993
I20260812 00:12:35.482971 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 16350
2026-08-12T00:12:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (655 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T00:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:35Z Disabled control of system clock
I20260812 00:12:35.517943 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42461
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:12:35.600405 17250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.600620 17250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.600642 17250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:35.601874 17250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:35.601917 17250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:35.601930 17250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:35.601943 17250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:35.601955 17250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:35.603338 17250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:35.603606 17250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:35.603915 17250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:35.609006 17257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.608971 17258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.608996 17250 server_base.cc:1034] running on GCE node
W20260812 00:12:35.608978 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.609635 17250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:35.609946 17250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:35.611089 17250 hybrid_clock.cc:648] HybridClock initialized: now 1786493555611078 us; error 29 us; skew 500 ppm
I20260812 00:12:35.612402 17250 webserver.cc:460] Webserver started at http://127.31.227.62:43149/ using document root <none> and password file <none>
I20260812 00:12:35.612653 17250 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:35.612702 17250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:35.612843 17250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:35.614013 17250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "5892843900a248378169b680b5741c13"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "e542c0274f9d69fe1c0c0bba3488d360"
server_key_iv: "7d41f369448dfd43afcc3075931c872f"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.614383 17250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "5892843900a248378169b680b5741c13"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "e542c0274f9d69fe1c0c0bba3488d360"
server_key_iv: "7d41f369448dfd43afcc3075931c872f"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.615713 17250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:35.616551 17270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.616740 17250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:35.616851 17250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "5892843900a248378169b680b5741c13"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "e542c0274f9d69fe1c0c0bba3488d360"
server_key_iv: "7d41f369448dfd43afcc3075931c872f"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.616928 17250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:35.620298 17250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:35.620560 17250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:35.620707 17250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:35.624759 17250 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42461
I20260812 00:12:35.624790 17368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42461 every 8 connection(s)
I20260812 00:12:35.625111 17250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:35.625828 17370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:35.628551 17370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13: Bootstrap starting.
I20260812 00:12:35.629132 17370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:35.629436 17370 log.cc:824] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:35.630177 17370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13: No bootstrap required, opened a new log
I20260812 00:12:35.631851 17370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892843900a248378169b680b5741c13" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42461 } }
I20260812 00:12:35.631987 17370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:35.632009 17370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5892843900a248378169b680b5741c13, State: Initialized, Role: FOLLOWER
I20260812 00:12:35.632180 17370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892843900a248378169b680b5741c13" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42461 } }
I20260812 00:12:35.632243 17370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:35.632264 17370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:35.632318 17370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:35.632575 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 17250
I20260812 00:12:35.632688 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:35.632906 32652 external_mini_cluster.cc:1428] Setting key cf68ea0d65b743d4362621901ea2f94a
I20260812 00:12:35.632954 17370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892843900a248378169b680b5741c13" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42461 } }
I20260812 00:12:35.633101 17370 leader_election.cc:302] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5892843900a248378169b680b5741c13; no voters:
I20260812 00:12:35.633270 17370 leader_election.cc:288] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:35.633323 17378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:35.633574 17378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [term 1 LEADER]: Becoming Leader. State: Replica: 5892843900a248378169b680b5741c13, State: Running, Role: LEADER
I20260812 00:12:35.633699 17378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892843900a248378169b680b5741c13" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42461 } }
I20260812 00:12:35.633713 17370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:35.634074 17382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5892843900a248378169b680b5741c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892843900a248378169b680b5741c13" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42461 } } }
I20260812 00:12:35.634156 17382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:35.634407 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5892843900a248378169b680b5741c13. Latest consensus state: current_term: 1 leader_uuid: "5892843900a248378169b680b5741c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892843900a248378169b680b5741c13" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42461 } } }
I20260812 00:12:35.634500 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:35.635279 17390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:35.635799 17390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:35.637188 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.633387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.31.227.62:42461)
0812 00:12:35.633689 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.633698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.635500 (+ 1802us) server_negotiation.cc:407] Connection header received
0812 00:12:35.635678 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.635686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.635735 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.635943 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.636627 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.636630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.636631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:35.636635 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.636677 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:35.636691 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.636767 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.636942 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.637006 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":170,"threads_started":1}
I20260812 00:12:35.637995 17390 catalog_manager.cc:1347] Generated new cluster ID: 056eb0f464dd474f9faf6225f45efb65
I20260812 00:12:35.638036 17390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:35.658834 17390 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:35.659029 17390 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:35.663842 17390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5892843900a248378169b680b5741c13: Generated new TSK 0
I20260812 00:12:35.664048 17390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:35.667759 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42461
--builtin_ntp_servers=127.31.227.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:12:35.764155 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.764360 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.764384 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:35.765573 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:35.765614 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:35.765640 17417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:35.766985 17417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:35.767241 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:35.767568 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:35.773419 17427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.773425 17426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.773569 17417 server_base.cc:1034] running on GCE node
W20260812 00:12:35.773768 17431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.774044 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:35.774319 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:35.775485 17417 hybrid_clock.cc:648] HybridClock initialized: now 1786493555775461 us; error 41 us; skew 500 ppm
I20260812 00:12:35.776769 17417 webserver.cc:460] Webserver started at http://127.31.227.1:33027/ using document root <none> and password file <none>
I20260812 00:12:35.777024 17417 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:35.777100 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:35.777262 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:35.778434 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "1c0fac87ff2f42bfb1070c630796526e"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "9571300b7742e415f21c32dff1d48cef"
server_key_iv: "1d226abb47da0c16a76df67f1b78a65d"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.778813 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "1c0fac87ff2f42bfb1070c630796526e"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "9571300b7742e415f21c32dff1d48cef"
server_key_iv: "1d226abb47da0c16a76df67f1b78a65d"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.780177 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:12:35.780942 17442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.781148 17417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:35.781250 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "1c0fac87ff2f42bfb1070c630796526e"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "9571300b7742e415f21c32dff1d48cef"
server_key_iv: "1d226abb47da0c16a76df67f1b78a65d"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.781332 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:35.784703 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:35.784977 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:35.785131 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:35.785342 17417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:35.785718 17417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:35.785772 17417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.785812 17417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:35.785838 17417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.791724 17417 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:37415
I20260812 00:12:35.791819 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:37415 every 8 connection(s)
I20260812 00:12:35.792167 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:35.794243 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 17417
I20260812 00:12:35.794345 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:35.794553 32652 external_mini_cluster.cc:1428] Setting key bf5b1a215d68ce3fd83618f5dbfea6c5
I20260812 00:12:35.795370 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.792878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38909 (local address 127.31.227.62:42461)
0812 00:12:35.792960 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.792963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.793173 (+ 210us) server_negotiation.cc:407] Connection header received
0812 00:12:35.793708 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.793709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.793716 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.793768 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.795090 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.795091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.795092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:35.795097 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.795114 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:35.795118 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.795184 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.795277 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.795294 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:12:35.795572 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.792841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42461 (local address 127.31.227.1:38909)
0812 00:12:35.793053 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.793086 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.793622 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.793819 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.793822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.793831 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.794923 (+ 1092us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:35.794924 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.794960 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:35.795004 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.795199 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.795202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.795205 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.795278 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.795342 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260812 00:12:35.795729 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42461
--builtin_ntp_servers=127.31.227.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 00:12:35.797660 17621 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42461
I20260812 00:12:35.797744 17621 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:35.797928 17621 heartbeater.cc:511] Master 127.31.227.62:42461 requested a full tablet report, sending...
I20260812 00:12:35.798466 17302 ts_manager.cc:194] Registered new tserver with Master: 1c0fac87ff2f42bfb1070c630796526e (127.31.227.1:37415)
W20260812 00:12:35.880947 17624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.881182 17624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.881222 17624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:35.882493 17624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:35.882555 17624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:35.882602 17624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:35.884043 17624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:35.884313 17624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:35.884627 17624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:35.890316 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.890336 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:35.890329 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:35.890727 17624 server_base.cc:1034] running on GCE node
I20260812 00:12:35.890988 17624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:35.891237 17624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:35.892382 17624 hybrid_clock.cc:648] HybridClock initialized: now 1786493555892370 us; error 35 us; skew 500 ppm
I20260812 00:12:35.893718 17624 webserver.cc:460] Webserver started at http://127.31.227.2:39917/ using document root <none> and password file <none>
I20260812 00:12:35.893975 17624 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:35.894021 17624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:35.894196 17624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:35.895272 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "2006f09c765c41a58f084bcd6e2a11bc"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "2c41a7016383b644c843ff209b0bd6b1"
server_key_iv: "88ba3023e07b95812a4d7eac8cb14c84"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.895630 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "2006f09c765c41a58f084bcd6e2a11bc"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "2c41a7016383b644c843ff209b0bd6b1"
server_key_iv: "88ba3023e07b95812a4d7eac8cb14c84"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.896947 17624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:35.897940 17645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.898106 17624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:35.898214 17624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "2006f09c765c41a58f084bcd6e2a11bc"
format_stamp: "Formatted at 2026-08-12 00:12:35 on dist-test-slave-zr1t"
server_key: "2c41a7016383b644c843ff209b0bd6b1"
server_key_iv: "88ba3023e07b95812a4d7eac8cb14c84"
server_key_version: "encryptionkey@0"
I20260812 00:12:35.898289 17624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:35.901914 17624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:35.902179 17624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:35.902362 17624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:35.902575 17624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:35.902894 17624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:35.902956 17624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.903000 17624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:35.903028 17624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:35.909106 17624 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:34065
I20260812 00:12:35.909159 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:34065 every 8 connection(s)
I20260812 00:12:35.909483 17624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:35.911633 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 17624
I20260812 00:12:35.911758 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:35.911991 32652 external_mini_cluster.cc:1428] Setting key 066b8d2b49a99c6ee269d50ab121fc9b
I20260812 00:12:35.912987 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.910287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:47387 (local address 127.31.227.62:42461)
0812 00:12:35.910378 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.910381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.910567 (+ 186us) server_negotiation.cc:407] Connection header received
0812 00:12:35.911052 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.911054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.911060 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.911121 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.912623 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.912624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.912624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:35.912628 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.912644 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:35.912649 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.912747 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.912888 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.912915 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 00:12:35.913170 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.910247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42461 (local address 127.31.227.2:47387)
0812 00:12:35.910437 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.910472 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.910942 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.911191 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.911194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.911202 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.912451 (+ 1249us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:35.912452 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.912490 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:35.912526 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.912773 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.912782 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.912785 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.912882 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.912951 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260812 00:12:35.913200 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42461
--builtin_ntp_servers=127.31.227.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 00:12:35.913700 17859 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42461
I20260812 00:12:35.913780 17859 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:35.913941 17859 heartbeater.cc:511] Master 127.31.227.62:42461 requested a full tablet report, sending...
I20260812 00:12:35.914319 17302 ts_manager.cc:194] Registered new tserver with Master: 2006f09c765c41a58f084bcd6e2a11bc (127.31.227.2:34065)
W20260812 00:12:35.998766 17863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:35.999017 17863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:35.999056 17863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:36.000272 17863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:36.000334 17863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:36.000378 17863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:36.001988 17863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:36.002261 17863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:36.002588 17863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:36.007879 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:36.007848 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:36.008030 17863 server_base.cc:1034] running on GCE node
W20260812 00:12:36.007858 17877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:36.008286 17863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:36.008486 17863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:36.009681 17863 hybrid_clock.cc:648] HybridClock initialized: now 1786493556009662 us; error 36 us; skew 500 ppm
I20260812 00:12:36.011004 17863 webserver.cc:460] Webserver started at http://127.31.227.3:46695/ using document root <none> and password file <none>
I20260812 00:12:36.011245 17863 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:36.011293 17863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:36.011473 17863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:36.012606 17863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "5c42c33d82e34f92a71de31dc8b2eeef"
format_stamp: "Formatted at 2026-08-12 00:12:36 on dist-test-slave-zr1t"
server_key: "3e1813d0e7d8fd71e3c2570e6fd8bb1b"
server_key_iv: "8f95910f0025aaffc04ee96d7032f552"
server_key_version: "encryptionkey@0"
I20260812 00:12:36.012972 17863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "5c42c33d82e34f92a71de31dc8b2eeef"
format_stamp: "Formatted at 2026-08-12 00:12:36 on dist-test-slave-zr1t"
server_key: "3e1813d0e7d8fd71e3c2570e6fd8bb1b"
server_key_iv: "8f95910f0025aaffc04ee96d7032f552"
server_key_version: "encryptionkey@0"
I20260812 00:12:36.014421 17863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:36.015168 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:36.015347 17863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:36.015437 17863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "5c42c33d82e34f92a71de31dc8b2eeef"
format_stamp: "Formatted at 2026-08-12 00:12:36 on dist-test-slave-zr1t"
server_key: "3e1813d0e7d8fd71e3c2570e6fd8bb1b"
server_key_iv: "8f95910f0025aaffc04ee96d7032f552"
server_key_version: "encryptionkey@0"
I20260812 00:12:36.015512 17863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:36.019018 17863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:36.019394 17863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:36.019507 17863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:36.019690 17863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:36.019997 17863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:36.020031 17863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:36.020051 17863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:36.020090 17863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:36.026373 17863 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:35673
I20260812 00:12:36.026425 18098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:35673 every 8 connection(s)
I20260812 00:12:36.026744 17863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:36.029471 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 17863
I20260812 00:12:36.029601 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:36.029809 32652 external_mini_cluster.cc:1428] Setting key 143239facdf2d75bc9e87d2445f29131
I20260812 00:12:36.029891 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:36.027460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:52415 (local address 127.31.227.62:42461)
0812 00:12:36.027519 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:36.027521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:36.027789 (+ 268us) server_negotiation.cc:407] Connection header received
0812 00:12:36.028331 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:36.028332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:36.028340 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:36.028398 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:36.029577 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:36.029579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:36.029579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:12:36.029584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:36.029600 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:12:36.029604 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:36.029676 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:36.029801 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 00:12:36.029820 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:12:36.030107 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:36.027452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42461 (local address 127.31.227.3:52415)
0812 00:12:36.027668 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:36.027702 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:36.028218 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:36.028483 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:36.028486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:36.028494 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:36.029386 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:12:36.029387 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:36.029414 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:12:36.029458 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:36.029674 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:36.029682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:36.029686 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:36.029809 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 00:12:36.029873 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260812 00:12:36.030530 18100 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42461
I20260812 00:12:36.030623 18100 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:36.030772 18100 heartbeater.cc:511] Master 127.31.227.62:42461 requested a full tablet report, sending...
I20260812 00:12:36.031124 17302 ts_manager.cc:194] Registered new tserver with Master: 5c42c33d82e34f92a71de31dc8b2eeef (127.31.227.3:35673)
I20260812 00:12:36.041992 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:12:36.043952 17385 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:36.043419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.31.227.62:42461)
0812 00:12:36.043484 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:36.043486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:36.043633 (+ 147us) server_negotiation.cc:407] Connection header received
0812 00:12:36.043678 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:36.043679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:36.043684 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:36.043732 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:36.043870 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43586: BlockingRecv error: recv got EOF from 127.0.0.1:43586 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:12:36.043970 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 17417
I20260812 00:12:36.058666 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 17624
I20260812 00:12:36.073258 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 17863
I20260812 00:12:36.082710 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 17250
2026-08-12T00:12:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (597 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:36 dist-test-slave-zr1t krb5kdc[18118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:36 dist-test-slave-zr1t krb5kdc[18118](info): set up 2 sockets
Aug 12 00:12:36 dist-test-slave-zr1t krb5kdc[18118](info): commencing operation
krb5kdc: starting...
W20260812 00:12:37.395888 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:37 dist-test-slave-zr1t krb5kdc[18118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:12:37.521713 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45065
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:37.607301 18144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:37.607519 18144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:37.607542 18144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:37.608759 18144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:37.608803 18144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:37.608816 18144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:37.608866 18144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:37.608884 18144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:37.610389 18144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:45065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:37.610662 18144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:37.610899 18144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:37.615945 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:37.615938 18156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:37.615916 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:37.615957 18144 server_base.cc:1034] running on GCE node
I20260812 00:12:37.616336 18144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:37.616593 18144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:37.617733 18144 hybrid_clock.cc:648] HybridClock initialized: now 1786493557617720 us; error 29 us; skew 500 ppm
Aug 12 00:12:37 dist-test-slave-zr1t krb5kdc[18118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:37.620301 18144 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:12:37.620731 18144 webserver.cc:460] Webserver started at http://127.31.227.62:39877/ using document root <none> and password file <none>
I20260812 00:12:37.620988 18144 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:37.621037 18144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:37.621194 18144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:37.622339 18144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "dae3cf2ff3984095a8dfc7d4dff12503"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "2d8a3dd6828220c35f5f9a157cff0ddd"
server_key_iv: "941acc6922bd3b7d9f853ac521c2f2db"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.622706 18144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "dae3cf2ff3984095a8dfc7d4dff12503"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "2d8a3dd6828220c35f5f9a157cff0ddd"
server_key_iv: "941acc6922bd3b7d9f853ac521c2f2db"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.624025 18144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:37.624751 18175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:37.624929 18144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:37.625036 18144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "dae3cf2ff3984095a8dfc7d4dff12503"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "2d8a3dd6828220c35f5f9a157cff0ddd"
server_key_iv: "941acc6922bd3b7d9f853ac521c2f2db"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.625118 18144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:37.645171 18144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:37.647712 18144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:37.648061 18144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:37.652477 18144 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:45065
I20260812 00:12:37.652549 18268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:45065 every 8 connection(s)
I20260812 00:12:37.652925 18144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:37.654009 18271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:37.656996 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503: Bootstrap starting.
I20260812 00:12:37.657588 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 18144
I20260812 00:12:37.657706 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:12:37.657706 18271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:37.657886 32652 external_mini_cluster.cc:1428] Setting key 07a017fca8a80ae97575b03f56d527f7
I20260812 00:12:37.658033 18271 log.cc:824] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:37.658874 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503: No bootstrap required, opened a new log
I20260812 00:12:37.660956 18271 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45065 } }
I20260812 00:12:37.661123 18271 raft_consensus.cc:374] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:37.661172 18271 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae3cf2ff3984095a8dfc7d4dff12503, State: Initialized, Role: FOLLOWER
I20260812 00:12:37.661342 18271 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45065 } }
I20260812 00:12:37.661454 18271 raft_consensus.cc:388] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:37.661500 18271 raft_consensus.cc:482] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:37.661579 18271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:37.662142 18271 raft_consensus.cc:504] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45065 } }
I20260812 00:12:37.662267 18271 leader_election.cc:302] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dae3cf2ff3984095a8dfc7d4dff12503; no voters:
I20260812 00:12:37.662488 18271 leader_election.cc:288] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:37.662513 18277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:37.662770 18271 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:37.662789 18277 raft_consensus.cc:686] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [term 1 LEADER]: Becoming Leader. State: Replica: dae3cf2ff3984095a8dfc7d4dff12503, State: Running, Role: LEADER
I20260812 00:12:37.662955 18277 consensus_queue.cc:237] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45065 } }
I20260812 00:12:37.663344 18277 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae3cf2ff3984095a8dfc7d4dff12503. Latest consensus state: current_term: 1 leader_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45065 } } }
I20260812 00:12:37.663357 18280 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae3cf2ff3984095a8dfc7d4dff12503" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45065 } } }
I20260812 00:12:37.663448 18277 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:37.663462 18280 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:37.663676 18293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:37.664238 18293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:12:37 dist-test-slave-zr1t krb5kdc[18118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:37.665544 18307 mvcc.cc:205] Tried to move back new op lower bound from 7317477612196388864 to 7317477612187803648. Current Snapshot: MvccSnapshot[applied={T|T < 7317477612196388864}]
I20260812 00:12:37.665954 18293 catalog_manager.cc:1347] Generated new cluster ID: a2349ea1c46e43aaa8debd5668edf873
I20260812 00:12:37.665992 18293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:37.668901 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:37.658473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.31.227.62:45065)
0812 00:12:37.658654 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:37.658665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:37.658699 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:12:37.659214 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:37.659221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:37.659270 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:37.659416 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:37.660066 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:37.660953 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:37.661936 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:37.662469 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:37.662588 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:37.665537 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:37.665541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:37.665542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:37.665556 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:37.667373 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:37.667755 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:37.667758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:37.667759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:37.667846 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:37.668056 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:37.668057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:37.668058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:37.668290 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:37.668469 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:37.668551 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:12:37.668695 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260812 00:12:37.677907 18293 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:37.678418 18293 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:37.687021 18293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503: Generated new TSK 0
I20260812 00:12:37.687212 18293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:12:37.723716 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45065
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:37.809685 18313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:37.809885 18313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:37.809909 18313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:37.811048 18313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:37.811091 18313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:37.811118 18313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:12:37.812680 18313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:37.812929 18313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:37.813160 18313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:37.818547 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:37.818540 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:37.818540 18320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:37.818881 18313 server_base.cc:1034] running on GCE node
I20260812 00:12:37.819090 18313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:37.819351 18313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:37.820501 18313 hybrid_clock.cc:648] HybridClock initialized: now 1786493557820474 us; error 38 us; skew 500 ppm
Aug 12 00:12:37 dist-test-slave-zr1t krb5kdc[18118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:37.823179 18313 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:12:37.823669 18313 webserver.cc:460] Webserver started at http://127.31.227.1:36227/ using document root <none> and password file <none>
I20260812 00:12:37.823897 18313 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:37.823941 18313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:37.824090 18313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:37.825161 18313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "5482ecc2dbf7457a83e56ac0844c6b29"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "c36a7e88e33b4d0b18478b488e9258e4"
server_key_iv: "c347d6ffdc4234fee174015f5b243394"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.825583 18313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "5482ecc2dbf7457a83e56ac0844c6b29"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "c36a7e88e33b4d0b18478b488e9258e4"
server_key_iv: "c347d6ffdc4234fee174015f5b243394"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.826937 18313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:37.827692 18339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:37.827853 18313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:37.827963 18313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "5482ecc2dbf7457a83e56ac0844c6b29"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "c36a7e88e33b4d0b18478b488e9258e4"
server_key_iv: "c347d6ffdc4234fee174015f5b243394"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.828048 18313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:37.848099 18313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:37.850682 18313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:37.850857 18313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:37.851123 18313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:37.851467 18313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:37.851521 18313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:37.851557 18313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:37.851600 18313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:37.857848 18313 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:41873
I20260812 00:12:37.857933 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:41873 every 8 connection(s)
I20260812 00:12:37.858244 18313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:37.859858 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 18313
I20260812 00:12:37.859972 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:37.860179 32652 external_mini_cluster.cc:1428] Setting key e94054a2c9116721326da162a4b872ce
Aug 12 00:12:37 dist-test-slave-zr1t krb5kdc[18118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:37.868575 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:37.858964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:35339 (local address 127.31.227.62:45065)
0812 00:12:37.859015 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:37.859017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:37.859254 (+ 237us) server_negotiation.cc:407] Connection header received
0812 00:12:37.859889 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:37.859891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:37.859905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:37.859962 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:37.861138 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:37.861702 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:37.862734 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:37.863023 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:37.863110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:37.865581 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:37.865582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:37.865583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:37.865591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:37.867109 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:37.867548 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:37.867550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:37.867550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:37.867617 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:37.867882 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:37.867883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:37.867883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:37.868009 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:37.868088 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:37.868380 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 00:12:37.868479 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:12:37.868767 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:37.858943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45065 (local address 127.31.227.1:35339)
0812 00:12:37.859125 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:37.859163 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:37.859757 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:37.860037 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:37.860042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:37.860802 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:37.861032 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:37.861038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:37.861798 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:37.861800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:37.862635 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:37.862637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:37.863128 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:37.863129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:37.863271 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:37.863934 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:37.863943 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:37.865451 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:37.867219 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:37.867228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:37.867230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:37.867466 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:37.867687 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:37.867688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:37.867688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:37.867808 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:37.868080 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:37.868088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:37.868239 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 00:12:37.868352 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:12:37.868518 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
I20260812 00:12:37.869256 18542 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45065
I20260812 00:12:37.869374 18542 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:37.869612 18542 heartbeater.cc:511] Master 127.31.227.62:45065 requested a full tablet report, sending...
I20260812 00:12:37.870165 18201 ts_manager.cc:194] Registered new tserver with Master: 5482ecc2dbf7457a83e56ac0844c6b29 (127.31.227.1:41873)
I20260812 00:12:37.870900 18201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:35339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:12:37.888021 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45065
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:37.975138 18548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:37.975355 18548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:37.975381 18548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:37.976475 18548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:37.976511 18548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:37.976537 18548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:12:37.977991 18548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:37.978194 18548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:37.978384 18548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:37.983762 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:37.983805 18548 server_base.cc:1034] running on GCE node
W20260812 00:12:37.983832 18559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:37.983948 18557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:37.984148 18548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:37.984407 18548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:37.985598 18548 hybrid_clock.cc:648] HybridClock initialized: now 1786493557985594 us; error 37 us; skew 500 ppm
Aug 12 00:12:37 dist-test-slave-zr1t krb5kdc[18118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:37.988170 18548 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:12:37.988644 18548 webserver.cc:460] Webserver started at http://127.31.227.2:34329/ using document root <none> and password file <none>
I20260812 00:12:37.988907 18548 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:37.988978 18548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:37.989133 18548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:37.990281 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "f30e408bf6354d18b9145578025ee88f"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "b89fa509830770d0147f740c46103523"
server_key_iv: "bb891df85310e1899c617235b95dde26"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.990670 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "f30e408bf6354d18b9145578025ee88f"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "b89fa509830770d0147f740c46103523"
server_key_iv: "bb891df85310e1899c617235b95dde26"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.992033 18548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:37.992770 18583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:37.992959 18548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:12:37.993070 18548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "f30e408bf6354d18b9145578025ee88f"
format_stamp: "Formatted at 2026-08-12 00:12:37 on dist-test-slave-zr1t"
server_key: "b89fa509830770d0147f740c46103523"
server_key_iv: "bb891df85310e1899c617235b95dde26"
server_key_version: "encryptionkey@0"
I20260812 00:12:37.993146 18548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:38.030439 18548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:38.033167 18548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:38.033353 18548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:38.033689 18548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:38.034040 18548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:38.034096 18548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:38.034138 18548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:38.034166 18548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:38.040910 18548 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:39825
I20260812 00:12:38.040956 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:39825 every 8 connection(s)
I20260812 00:12:38.041308 18548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:38.044556 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 18548
I20260812 00:12:38.044695 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:38.044965 32652 external_mini_cluster.cc:1428] Setting key 92b58f23a92d5afa3e555e266c3a1f09
Aug 12 00:12:38 dist-test-slave-zr1t krb5kdc[18118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493557, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:38.051605 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.042134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60993 (local address 127.31.227.62:45065)
0812 00:12:38.042184 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.042186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.042466 (+ 280us) server_negotiation.cc:407] Connection header received
0812 00:12:38.043092 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.043094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.043108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.043174 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:38.044192 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.044745 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.045986 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.046189 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.046269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.048660 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:38.048662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:38.048663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:38.048671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:38.050123 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:38.050552 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:38.050554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:38.050554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:38.050621 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:38.050892 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:38.050893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:38.050893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:38.051028 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:38.051107 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.051407 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.051509 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:12:38.051826 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.042119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45065 (local address 127.31.227.2:60993)
0812 00:12:38.042333 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.042372 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.042983 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.043265 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.043268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.043866 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:38.044076 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.044081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.044848 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.044850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.045895 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.045898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.046278 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.046279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.046409 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.046992 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:38.047001 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:38.048552 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:38.050214 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:38.050223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:38.050225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:38.050463 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:38.050698 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:38.050699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:38.050699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:38.050817 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:38.051097 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:38.051106 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:38.051261 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.051369 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.051557 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260812 00:12:38.052376 18785 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45065
I20260812 00:12:38.052507 18785 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:38.052711 18785 heartbeater.cc:511] Master 127.31.227.62:45065 requested a full tablet report, sending...
I20260812 00:12:38.053194 18201 ts_manager.cc:194] Registered new tserver with Master: f30e408bf6354d18b9145578025ee88f (127.31.227.2:39825)
I20260812 00:12:38.053659 18201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:60993
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:12:38.073302 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45065
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:38.160784 18798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:38.160993 18798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:38.161036 18798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:38.162220 18798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:38.162264 18798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:38.162305 18798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:12:38.163935 18798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:12:38.164134 18798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:38.164367 18798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:38.169624 18810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:38.169634 18805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:38.169624 18804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:38.169775 18798 server_base.cc:1034] running on GCE node
I20260812 00:12:38.170104 18798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:38.170401 18798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:38.171568 18798 hybrid_clock.cc:648] HybridClock initialized: now 1786493558171556 us; error 31 us; skew 500 ppm
Aug 12 00:12:38 dist-test-slave-zr1t krb5kdc[18118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493558, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:38.174201 18798 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:12:38.174667 18798 webserver.cc:460] Webserver started at http://127.31.227.3:34017/ using document root <none> and password file <none>
I20260812 00:12:38.174850 18798 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:38.174898 18798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:38.174985 18798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:38.176019 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "3c5445b900924d83b4064e60cef7eafe"
format_stamp: "Formatted at 2026-08-12 00:12:38 on dist-test-slave-zr1t"
server_key: "a2b68032511e987778f38b7b00a3a016"
server_key_iv: "810318e224d08b3f8f21742acc81e16b"
server_key_version: "encryptionkey@0"
I20260812 00:12:38.176324 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "3c5445b900924d83b4064e60cef7eafe"
format_stamp: "Formatted at 2026-08-12 00:12:38 on dist-test-slave-zr1t"
server_key: "a2b68032511e987778f38b7b00a3a016"
server_key_iv: "810318e224d08b3f8f21742acc81e16b"
server_key_version: "encryptionkey@0"
I20260812 00:12:38.177645 18798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:12:38.178377 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:38.178536 18798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:12:38.178622 18798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "3c5445b900924d83b4064e60cef7eafe"
format_stamp: "Formatted at 2026-08-12 00:12:38 on dist-test-slave-zr1t"
server_key: "a2b68032511e987778f38b7b00a3a016"
server_key_iv: "810318e224d08b3f8f21742acc81e16b"
server_key_version: "encryptionkey@0"
I20260812 00:12:38.178680 18798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:38.187767 18798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:38.189906 18798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:38.190058 18798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:38.190285 18798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:38.190621 18798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:38.190675 18798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:38.190714 18798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:38.190747 18798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:38.196669 18798 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:37233
I20260812 00:12:38.196728 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:37233 every 8 connection(s)
I20260812 00:12:38.197106 18798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:38.199191 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 18798
I20260812 00:12:38.199270 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:38.199465 32652 external_mini_cluster.cc:1428] Setting key 889caa187b34b25d52d9a1512a898a3c
Aug 12 00:12:38 dist-test-slave-zr1t krb5kdc[18118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493558, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:12:38.207216 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.197886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:55647 (local address 127.31.227.62:45065)
0812 00:12:38.197933 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.197935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.198174 (+ 239us) server_negotiation.cc:407] Connection header received
0812 00:12:38.198730 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.198732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.198745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.198810 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:38.199942 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.200587 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.201642 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.201848 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.201933 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.204168 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:38.204170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:38.204171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:38.204180 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:38.205802 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:38.206220 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:38.206222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:38.206222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:38.206288 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:38.206548 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:38.206549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:38.206550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:38.206673 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:38.206748 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.207042 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.207123 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:12:38.207415 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.197849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45065 (local address 127.31.227.3:55647)
0812 00:12:38.198052 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.198089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.198623 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.198895 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.198899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.199590 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:38.199868 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.199875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.200690 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.200691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.201508 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.201510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.201947 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.201948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.202080 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.202650 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:38.202659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:38.204060 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:38.205904 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:38.205912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:38.205914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:38.206140 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:38.206356 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:38.206357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:38.206357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:38.206473 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:38.206738 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:38.206745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:38.206898 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.207006 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.207172 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260812 00:12:38.207931 19022 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45065
I20260812 00:12:38.208039 19022 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:38.208230 19022 heartbeater.cc:511] Master 127.31.227.62:45065 requested a full tablet report, sending...
I20260812 00:12:38.208643 18201 ts_manager.cc:194] Registered new tserver with Master: 3c5445b900924d83b4064e60cef7eafe (127.31.227.3:37233)
I20260812 00:12:38.208985 18201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:55647
I20260812 00:12:38.211175 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:38.218397 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.212486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.31.227.62:45065)
0812 00:12:38.212560 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.212563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.212665 (+ 102us) server_negotiation.cc:407] Connection header received
0812 00:12:38.212742 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.212743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.212753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.212809 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:38.213357 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.213905 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.214649 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.214849 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.214929 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.216083 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:38.216084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:38.216085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:38.216092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:38.217220 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:38.217589 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:38.217590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:38.217591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:38.217649 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:38.217851 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:38.217852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:38.217852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:38.218004 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:38.218109 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.218216 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.218297 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:12:38.219929 18201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:38.221475 18201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:38.228053 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.223799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:39825 (local address 127.0.0.1:42702)
0812 00:12:38.223980 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.224035 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.224111 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.224518 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.224522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.224525 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:38.224642 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.224645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.225766 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.225767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.227029 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.227032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.227749 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.227750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.227827 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.227842 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.227914 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.227971 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260812 00:12:38.228062 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.224081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.31.227.2:39825)
0812 00:12:38.224341 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.224344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.224353 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:38.224391 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.224393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.224410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.224737 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:38.224820 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.225664 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.227130 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.227659 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.227749 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.227823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.227939 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.227988 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260812 00:12:38.228274 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.223801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:37233 (local address 127.0.0.1:42476)
0812 00:12:38.224137 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.224150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.224209 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.224520 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.224524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.224530 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:38.224612 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.224614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.226085 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.226086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.227135 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.227137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.227991 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.227992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.228057 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.228061 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.228094 (+ 33us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.228136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":69,"threads_started":1}
I20260812 00:12:38.228438 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.223939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.31.227.3:37233)
0812 00:12:38.224296 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.224302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.224314 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:12:38.224360 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.224363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.224380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.224503 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:38.224705 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.225963 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.227215 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.227892 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.228186 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.228267 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.228295 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.228343 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":96,"threads_started":1}
I20260812 00:12:38.228493 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.224005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41873 (local address 127.0.0.1:38686)
0812 00:12:38.224458 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.224468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.224533 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.224813 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.224814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.224816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:38.224852 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.224854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.226118 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.226119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.227018 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.227020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.228025 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.228026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.228076 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.228079 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.228108 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.228136 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260812 00:12:38.228843 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.224276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.31.227.1:41873)
0812 00:12:38.224454 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.224459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.224591 (+ 132us) server_negotiation.cc:407] Connection header received
0812 00:12:38.224636 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.224638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.224655 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.224727 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:38.224931 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.226017 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.227128 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.227867 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.228584 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.228654 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.228687 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.228757 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260812 00:12:38.229317 18921 tablet_service.cc:1467] Processing CreateTablet for tablet e7b575a18d1440808ba333f3b9632025 (DEFAULT_TABLE table=test-table [id=f7071205ec7642fdb3f65be77fd5c51d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:38.229466 18662 tablet_service.cc:1467] Processing CreateTablet for tablet e7b575a18d1440808ba333f3b9632025 (DEFAULT_TABLE table=test-table [id=f7071205ec7642fdb3f65be77fd5c51d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:38.229496 18419 tablet_service.cc:1467] Processing CreateTablet for tablet e7b575a18d1440808ba333f3b9632025 (DEFAULT_TABLE table=test-table [id=f7071205ec7642fdb3f65be77fd5c51d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:38.229758 18921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b575a18d1440808ba333f3b9632025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:38.229823 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b575a18d1440808ba333f3b9632025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:38.229878 18662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b575a18d1440808ba333f3b9632025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:38.232911 19054 tablet_bootstrap.cc:492] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Bootstrap starting.
I20260812 00:12:38.233318 19057 tablet_bootstrap.cc:492] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: Bootstrap starting.
I20260812 00:12:38.233345 19060 tablet_bootstrap.cc:492] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Bootstrap starting.
I20260812 00:12:38.233700 19054 tablet_bootstrap.cc:654] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:38.234040 19060 tablet_bootstrap.cc:654] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:38.234040 19057 tablet_bootstrap.cc:654] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:38.234278 19054 log.cc:824] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:38.234658 19057 log.cc:824] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:38.234658 19060 log.cc:824] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:38.235091 19054 tablet_bootstrap.cc:492] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: No bootstrap required, opened a new log
I20260812 00:12:38.235167 19054 ts_tablet_manager.cc:1397] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:12:38.235747 19060 tablet_bootstrap.cc:492] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: No bootstrap required, opened a new log
I20260812 00:12:38.235821 19060 ts_tablet_manager.cc:1397] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:38.237280 19054 raft_consensus.cc:348] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.237460 19054 raft_consensus.cc:374] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:38.237543 19054 raft_consensus.cc:729] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e408bf6354d18b9145578025ee88f, State: Initialized, Role: FOLLOWER
I20260812 00:12:38.237675 19054 consensus_queue.cc:260] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.237824 19060 raft_consensus.cc:348] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.237994 19054 ts_tablet_manager.cc:1428] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:12:38.237996 19060 raft_consensus.cc:374] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:38.238039 19060 raft_consensus.cc:729] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5445b900924d83b4064e60cef7eafe, State: Initialized, Role: FOLLOWER
I20260812 00:12:38.238046 18785 heartbeater.cc:503] Master 127.31.227.62:45065 was elected leader, sending a full tablet report...
I20260812 00:12:38.238155 19060 consensus_queue.cc:260] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.238420 19060 ts_tablet_manager.cc:1428] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260812 00:12:38.238468 19022 heartbeater.cc:503] Master 127.31.227.62:45065 was elected leader, sending a full tablet report...
I20260812 00:12:38.238867 19057 tablet_bootstrap.cc:492] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: No bootstrap required, opened a new log
I20260812 00:12:38.238933 19057 ts_tablet_manager.cc:1397] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 00:12:38.240564 19057 raft_consensus.cc:348] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.240672 19057 raft_consensus.cc:374] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:38.240691 19057 raft_consensus.cc:729] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5482ecc2dbf7457a83e56ac0844c6b29, State: Initialized, Role: FOLLOWER
I20260812 00:12:38.240770 19057 consensus_queue.cc:260] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.241011 19057 ts_tablet_manager.cc:1428] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:38.241050 18542 heartbeater.cc:503] Master 127.31.227.62:45065 was elected leader, sending a full tablet report...
W20260812 00:12:38.292104 18790 tablet.cc:2369] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:38.359023 18543 tablet.cc:2369] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:38.430943 19069 raft_consensus.cc:482] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:38.431147 19069 raft_consensus.cc:504] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.431650 19069 leader_election.cc:288] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f30e408bf6354d18b9145578025ee88f (127.31.227.2:39825), 5482ecc2dbf7457a83e56ac0844c6b29 (127.31.227.1:41873)
I20260812 00:12:38.435047 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.431838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:39825 (local address 127.31.227.3:37269)
0812 00:12:38.431934 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.431947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.432031 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.432234 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.432235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.432237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:38.432321 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.432324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.433089 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.433090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.434102 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.434105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.434712 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.434714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.434816 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.434821 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.434906 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.434946 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":34}
I20260812 00:12:38.435065 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.431935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:37269 (local address 127.31.227.2:39825)
0812 00:12:38.432026 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.432028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.432041 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:12:38.432128 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.432130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.432143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.432221 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:38.432417 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.433008 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.434197 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.434626 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.434687 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.434695 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.434929 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.434981 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:12:38.435170 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.432130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58017 (local address 127.31.227.1:41873)
0812 00:12:38.432180 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.432182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.432280 (+ 98us) server_negotiation.cc:407] Connection header received
0812 00:12:38.432404 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.432406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.432419 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.432479 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:38.432671 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.433257 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.434308 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.434748 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.434830 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.434839 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.435037 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.435089 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:38.435191 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.432020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41873 (local address 127.31.227.3:58017)
0812 00:12:38.432196 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.432208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:38.432312 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:38.432522 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:38.432523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:38.432525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:38.432595 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.432597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.433356 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.433357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.434206 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:38.434208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.434840 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.434841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:38.434949 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.434953 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:38.435014 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:12:38.435053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260812 00:12:38.435570 18703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b575a18d1440808ba333f3b9632025" candidate_uuid: "3c5445b900924d83b4064e60cef7eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e408bf6354d18b9145578025ee88f" is_pre_election: true
I20260812 00:12:38.435658 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b575a18d1440808ba333f3b9632025" candidate_uuid: "3c5445b900924d83b4064e60cef7eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" is_pre_election: true
I20260812 00:12:38.435784 18465 raft_consensus.cc:2454] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5445b900924d83b4064e60cef7eafe in term 0.
I20260812 00:12:38.435784 18703 raft_consensus.cc:2454] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5445b900924d83b4064e60cef7eafe in term 0.
I20260812 00:12:38.436051 18827 leader_election.cc:302] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5445b900924d83b4064e60cef7eafe, f30e408bf6354d18b9145578025ee88f; no voters:
I20260812 00:12:38.436185 19069 raft_consensus.cc:2781] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:38.436249 19069 raft_consensus.cc:482] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:38.436285 19069 raft_consensus.cc:3037] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:38.437084 19069 raft_consensus.cc:504] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.437242 19069 leader_election.cc:288] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [CANDIDATE]: Term 1 election: Requested vote from peers f30e408bf6354d18b9145578025ee88f (127.31.227.2:39825), 5482ecc2dbf7457a83e56ac0844c6b29 (127.31.227.1:41873)
I20260812 00:12:38.437438 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b575a18d1440808ba333f3b9632025" candidate_uuid: "3c5445b900924d83b4064e60cef7eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5482ecc2dbf7457a83e56ac0844c6b29"
I20260812 00:12:38.437465 18703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b575a18d1440808ba333f3b9632025" candidate_uuid: "3c5445b900924d83b4064e60cef7eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e408bf6354d18b9145578025ee88f"
I20260812 00:12:38.437559 18465 raft_consensus.cc:3037] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:38.437640 18703 raft_consensus.cc:3037] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:38.438396 18465 raft_consensus.cc:2454] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5445b900924d83b4064e60cef7eafe in term 1.
I20260812 00:12:38.438580 18827 leader_election.cc:302] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5445b900924d83b4064e60cef7eafe, 5482ecc2dbf7457a83e56ac0844c6b29; no voters:
I20260812 00:12:38.438625 18703 raft_consensus.cc:2454] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5445b900924d83b4064e60cef7eafe in term 1.
I20260812 00:12:38.438691 19069 raft_consensus.cc:2781] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:38.438868 19069 raft_consensus.cc:686] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 1 LEADER]: Becoming Leader. State: Replica: 3c5445b900924d83b4064e60cef7eafe, State: Running, Role: LEADER
I20260812 00:12:38.439006 19069 consensus_queue.cc:237] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } }
I20260812 00:12:38.439917 18200 catalog_manager.cc:5495] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5445b900924d83b4064e60cef7eafe (127.31.227.3). New cstate: current_term: 1 leader_uuid: "3c5445b900924d83b4064e60cef7eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5445b900924d83b4064e60cef7eafe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 } health_report { overall_health: UNKNOWN } } }
W20260812 00:12:38.447700 19026 tablet.cc:2369] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:38.454836 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.451977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.31.227.3:37233)
0812 00:12:38.452045 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:38.452048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:38.452057 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:38.452102 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:38.452103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:38.452125 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:38.452223 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:38.452407 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.452948 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.453909 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:38.454147 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:38.454234 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:38.454432 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:38.454556 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:38.454666 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:38.454696 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 00:12:38.454718 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:12:38.456856 18703 raft_consensus.cc:1261] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 1 FOLLOWER]: Refusing update from remote peer 3c5445b900924d83b4064e60cef7eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:38.456856 18465 raft_consensus.cc:1261] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 1 FOLLOWER]: Refusing update from remote peer 3c5445b900924d83b4064e60cef7eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:38.457222 19069 consensus_queue.cc:1035] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f30e408bf6354d18b9145578025ee88f" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:38.457420 19072 consensus_queue.cc:1035] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5482ecc2dbf7457a83e56ac0844c6b29" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:38.459684 19079 mvcc.cc:205] Tried to move back new op lower bound from 7317477615435853824 to 7317477615366893568. Current Snapshot: MvccSnapshot[applied={T|T < 7317477615435853824}]
I20260812 00:12:38.459949 19083 mvcc.cc:205] Tried to move back new op lower bound from 7317477615435853824 to 7317477615366893568. Current Snapshot: MvccSnapshot[applied={T|T < 7317477615435853824}]
I20260812 00:12:38.460346 19080 mvcc.cc:205] Tried to move back new op lower bound from 7317477615435853824 to 7317477615366893568. Current Snapshot: MvccSnapshot[applied={T|T < 7317477615435853824}]
I20260812 00:12:38.462265 18201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:38.462410 18201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:38.463261 18201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dae3cf2ff3984095a8dfc7d4dff12503: Sending DeleteTablet for 3 replicas of tablet e7b575a18d1440808ba333f3b9632025
I20260812 00:12:38.463694 18419 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b575a18d1440808ba333f3b9632025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:38 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:38686
I20260812 00:12:38.463727 18921 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b575a18d1440808ba333f3b9632025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:38 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:42476
I20260812 00:12:38.463984 19096 tablet_replica.cc:330] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: stopping tablet replica
I20260812 00:12:38.464031 18662 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b575a18d1440808ba333f3b9632025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:38 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:42702
I20260812 00:12:38.464087 19096 raft_consensus.cc:2229] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:38.464172 19095 tablet_replica.cc:330] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29: stopping tablet replica
I20260812 00:12:38.464200 19096 raft_consensus.cc:2258] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:38.464340 19095 raft_consensus.cc:2229] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:38.464499 19095 raft_consensus.cc:2258] T e7b575a18d1440808ba333f3b9632025 P 5482ecc2dbf7457a83e56ac0844c6b29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:38.464659 19097 tablet_replica.cc:330] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: stopping tablet replica
I20260812 00:12:38.464761 19097 raft_consensus.cc:2229] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:38.464927 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 18313
I20260812 00:12:38.482625 19097 raft_consensus.cc:2258] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:38.483918 19097 ts_tablet_manager.cc:1905] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:38.485648 19097 ts_tablet_manager.cc:1918] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:38.485719 19097 log.cc:1196] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/e7b575a18d1440808ba333f3b9632025
I20260812 00:12:38.486012 19097 ts_tablet_manager.cc:1939] T e7b575a18d1440808ba333f3b9632025 P f30e408bf6354d18b9145578025ee88f: Deleting consensus metadata
I20260812 00:12:38.486501 18184 catalog_manager.cc:4841] TS f30e408bf6354d18b9145578025ee88f (127.31.227.2:39825): tablet e7b575a18d1440808ba333f3b9632025 (table test-table [id=f7071205ec7642fdb3f65be77fd5c51d]) successfully deleted
I20260812 00:12:38.492340 19096 ts_tablet_manager.cc:1905] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:38.494769 19096 ts_tablet_manager.cc:1918] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:38.494882 19096 log.cc:1196] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/e7b575a18d1440808ba333f3b9632025
I20260812 00:12:38.495242 19096 ts_tablet_manager.cc:1939] T e7b575a18d1440808ba333f3b9632025 P 3c5445b900924d83b4064e60cef7eafe: Deleting consensus metadata
I20260812 00:12:38.495792 18177 catalog_manager.cc:4841] TS 3c5445b900924d83b4064e60cef7eafe (127.31.227.3:37233): tablet e7b575a18d1440808ba333f3b9632025 (table test-table [id=f7071205ec7642fdb3f65be77fd5c51d]) successfully deleted
W20260812 00:12:38.497673 18184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:41873 (error 108)
I20260812 00:12:38.498323 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.498159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41873 (local address 127.0.0.1:38692)
0812 00:12:38.498227 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.498276 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 00:12:38.498416 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 18548
W20260812 00:12:38.505695 18184 catalog_manager.cc:4584] TS 5482ecc2dbf7457a83e56ac0844c6b29 (127.31.227.1:41873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7b575a18d1440808ba333f3b9632025: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41873: connect: Connection refused (error 111)
I20260812 00:12:38.511299 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 18798
I20260812 00:12:38.525626 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 18144
2026-08-12T00:12:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2445 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:38 dist-test-slave-zr1t krb5kdc[19103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:38 dist-test-slave-zr1t krb5kdc[19103](info): set up 2 sockets
Aug 12 00:12:38 dist-test-slave-zr1t krb5kdc[19103](info): commencing operation
krb5kdc: starting...
W20260812 00:12:39.807935 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:12:39 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/rangeradmin_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/rangeradmin_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/rangerlookup_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/rangerlookup_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
I20260812 00:12:39.990110 32652 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSkgTkG/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/postgres -l logfile start
2026-08-12 00:12:42.556 UTC [19159] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 00:12:42.556 UTC [19159] LOG: listening on IPv4 address "127.31.227.20", port 50295
2026-08-12 00:12:42.559 UTC [19159] LOG: listening on Unix socket "/tmp/.s.PGSQL.50295"
2026-08-12 00:12:42.596 UTC [19165] LOG: database system was shut down at 2026-08-12 00:12:41 UTC
2026-08-12 00:12:42.600 UTC [19159] LOG: database system is ready to accept connections
I20260812 00:12:43.777769 32652 mini_postgres.cc:96] Postgres bound to 50295
2026-08-12 00:12:43.786 UTC [19179] FATAL: database "slave" does not exist
127.31.227.20:50295 - accepting connections
I20260812 00:12:43.787818 32652 mini_ranger.cc:161] Starting Ranger...
I20260812 00:12:43.805235 32652 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 00:12:44.310752 32652 mini_ranger.cc:87] Created ranger Postgres database
I20260812 00:12:44.310844 32652 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-admin
2026-08-12 00:12:44,634 [I] DB FLAVOR :POSTGRES
2026-08-12 00:12:44,634 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:12:44,634 [I] Checking connection..
2026-08-12 00:12:44,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 00:12:44,905 [I] Checking connection passed.
2026-08-12 00:12:44,905 [I] --------- Verifying version history table ---------
2026-08-12 00:12:44,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:12:45,161 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:12:45,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:12:45,443 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:12:45,443 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 00:12:45,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 00:12:45.683 UTC [19293] WARNING: there is no transaction in progress
2026-08-12 00:12:45,693 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 00:12:45,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:12:45,941 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 00:12:45,942 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:12:45,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 00:12:46,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 00:12:46,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zr1t.c.gcp-upstream.internal','N') ;"
2026-08-12 00:12:46,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:12:46,976 [I] Table x_portal_user does not exist in database ranger
2026-08-12 00:12:46,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:12:47,246 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 00:12:47,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:12:47,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:12:47,795 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 00:12:47,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 00:12:48.774 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.792 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.810 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.826 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.843 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.862 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.918 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.925 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.939 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.949 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.960 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.975 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.984 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.990 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:48.997 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:49.004 UTC [19474] WARNING: there is no transaction in progress
2026-08-12 00:12:49,661 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 00:12:49,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 00:12:49,929 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 00:12:49,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:12:50,189 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 00:12:50,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:12:50,451 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 00:12:50,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:12:50,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:12:50,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zr1t.c.gcp-upstream.internal';"
2026-08-12 00:12:51,224 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 00:12:51,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 00:12:51,492 [I] DB_PATCHES have already been applied
I20260812 00:12:51.499042 32652 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSkgTkG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 00:12:51.499112 32652 mini_ranger.cc:193] Using host: 127.31.227.20
I20260812 00:12:51.503086 32652 mini_ranger.cc:227] Ranger admin URL: http://127.31.227.20:45117
Aug 12, 2026 12:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 12:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 12:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 12:12:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 12:12:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 00:12:52.927518 32652 mini_ranger.cc:160] Time spent starting Ranger: real 9.140s user 0.005s sys 0.007s
Aug 12, 2026 12:13:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 12:13:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 12:13:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 12:13:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 12:13:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 00:13:14.895808 32652 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/rangerkms_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/rangerkms_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.227.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.227.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 00:13:14 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493594, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 00:13:14.980871 32652 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 00:13:14.990260 32652 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 00:13:15.730082 32652 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 00:13:15.730175 32652 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms
I20260812 00:13:15.730193 32652 mini_ranger_kms.cc:227] Using postgres at 127.31.227.20:50295
2026-08-12 00:13:16,293 [I] DB FLAVOR :POSTGRES
2026-08-12 00:13:16,293 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:13:16,293 [I] Checking connection
2026-08-12 00:13:16,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 00:13:16,532 [I] connection success
2026-08-12 00:13:16,532 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 00:13:16,532 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 00:13:16,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 00:13:16,793 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 00:13:16,793 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:13:16,793 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 00:13:16,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:50295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 00:13:17,076 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 00:13:17.548812 32652 mini_ranger_kms.cc:310] Created kms service
I20260812 00:13:17.760288 32652 mini_ranger_kms.cc:326] Created kudu user
I20260812 00:13:17.802814 32652 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 00:13:17 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493594, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:18.200842 32652 mini_ranger_kms.cc:384] Added ranger policy
I20260812 00:13:18.200979 32652 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 00:13:18.201009 32652 mini_ranger_kms.cc:240] Using host: 127.31.227.20
I20260812 00:13:18.205435 32652 mini_ranger_kms.cc:276] Ranger KMS PID: 19792
I20260812 00:13:18.205590 32652 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.227.20:40947
Aug 12, 2026 12:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 12:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 12:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 12:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 00:13:20 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493600, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.227.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 00:13:20 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493600, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.227.20@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:21.077951 32652 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.097s user 0.015s sys 0.241s
I20260812 00:13:21.078058 32652 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 00:13:21.078135 32652 mini_ranger_kms.cc:401] 127.31.227.20:40947/kms/v1/keys
Aug 12 00:13:21 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493600, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.227.20@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
Aug 12 00:13:23 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:13:23.530771 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:35635
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:35635
--ranger_config_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:40947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:23.615276 19876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:23.615521 19876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:23.615561 19876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:23.616788 19876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:23.616854 19876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:23.616880 19876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:23.616910 19876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:23.616935 19876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:23.618649 19876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.227.20:40947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:35635
--ranger_config_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:13:23.618925 19876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:23.619158 19876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:23.624493 19895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:23.624624 19887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:23.624506 19884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:23.624742 19876 server_base.cc:1034] running on GCE node
I20260812 00:13:23.624993 19876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:23.625345 19876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:23.626504 19876 hybrid_clock.cc:648] HybridClock initialized: now 1786493603626481 us; error 45 us; skew 500 ppm
Aug 12 00:13:23 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493603, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:23.633309 19876 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:13:23.633867 19876 webserver.cc:460] Webserver started at http://127.31.227.62:40579/ using document root <none> and password file <none>
I20260812 00:13:23.634127 19876 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:23.634169 19876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:23.634310 19876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:13:23 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493603, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:23.680928 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "5c85e734ab3b4085a1f22d611eaf5d3c"
format_stamp: "Formatted at 2026-08-12 00:13:23 on dist-test-slave-zr1t"
server_key: "490df829c2b3eacc890d6ee421be3db8"
server_key_iv: "c0e925c49031dc3701f566d48473e23e"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:23.681375 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "5c85e734ab3b4085a1f22d611eaf5d3c"
format_stamp: "Formatted at 2026-08-12 00:13:23 on dist-test-slave-zr1t"
server_key: "490df829c2b3eacc890d6ee421be3db8"
server_key_iv: "c0e925c49031dc3701f566d48473e23e"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:23.683027 19876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:23.694818 19907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:23.695062 19876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:23.695168 19876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "5c85e734ab3b4085a1f22d611eaf5d3c"
format_stamp: "Formatted at 2026-08-12 00:13:23 on dist-test-slave-zr1t"
server_key: "490df829c2b3eacc890d6ee421be3db8"
server_key_iv: "c0e925c49031dc3701f566d48473e23e"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:23.695282 19876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:23.705314 19876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:23.707322 19876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:23.707485 19876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:23.712991 19876 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:35635
I20260812 00:13:23.713016 19995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:35635 every 8 connection(s)
I20260812 00:13:23.713416 19876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:23.714282 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:23.717453 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 19876
I20260812 00:13:23.717448 19996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c: Bootstrap starting.
I20260812 00:13:23.717599 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:13:23.717787 32652 external_mini_cluster.cc:1428] Setting key 6327d203e899c0e6a32744ce0b941792
I20260812 00:13:23.718221 19996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:23.718623 19996 log.cc:824] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:23.719482 19996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c: No bootstrap required, opened a new log
I20260812 00:13:23.722101 19996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35635 } }
I20260812 00:13:23.722306 19996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:23.722376 19996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c85e734ab3b4085a1f22d611eaf5d3c, State: Initialized, Role: FOLLOWER
I20260812 00:13:23.722551 19996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35635 } }
I20260812 00:13:23.722640 19996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:23.722669 19996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:23.723083 19996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:23.723851 19996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35635 } }
I20260812 00:13:23.723974 19996 leader_election.cc:302] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c85e734ab3b4085a1f22d611eaf5d3c; no voters:
I20260812 00:13:23.724179 19996 leader_election.cc:288] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:23.724243 20007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:23.724489 20007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [term 1 LEADER]: Becoming Leader. State: Replica: 5c85e734ab3b4085a1f22d611eaf5d3c, State: Running, Role: LEADER
I20260812 00:13:23.724529 19996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 00:13:23 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:13:23.725261 20007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35635 } }
I20260812 00:13:23.726649 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35635 } } }
I20260812 00:13:23.726742 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:23.727027 20009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c85e734ab3b4085a1f22d611eaf5d3c. Latest consensus state: current_term: 1 leader_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c85e734ab3b4085a1f22d611eaf5d3c" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 35635 } } }
I20260812 00:13:23.727130 20009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:23.727263 19996 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 00:13:23.727286 20028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:23.727815 20028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:23.736984 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:23.718317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.31.227.62:35635)
0812 00:13:23.718513 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:23.718523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:23.718557 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:13:23.719199 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:23.719207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:23.719261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:23.719417 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:23.720110 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:23.721001 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:23.721842 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:23.722273 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:23.722765 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:23.725363 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:23.725368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:23.725368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:23.725383 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:23.727705 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:23.728123 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:23.729595 (+ 1472us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:23.729596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:23.730241 (+ 645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:23.730543 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:23.730544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:23.730544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:23.731114 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:23.736388 (+ 5274us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:23.736504 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 00:13:23.736727 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260812 00:13:23.742096 20028 catalog_manager.cc:1347] Generated new cluster ID: 5c56eee5afd245829ecec001d28e59fe
I20260812 00:13:23.742156 20028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:23.760782 20028 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:23.761646 20028 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:23.783206 20028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c: Generated new TSK 0
I20260812 00:13:23.783598 20028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:13:24.546640 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:40947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35635
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:24.663857 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:24.664088 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:24.664129 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:24.665314 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:24.665390 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:24.665436 20078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:13:24.667004 20078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.227.20:40947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:13:24.667273 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:24.667552 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:24.676275 20093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:24.676496 20090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:24.676702 20089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:24.676847 20078 server_base.cc:1034] running on GCE node
I20260812 00:13:24.677122 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:24.677423 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:24.678606 20078 hybrid_clock.cc:648] HybridClock initialized: now 1786493604678580 us; error 39 us; skew 500 ppm
Aug 12 00:13:24 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:24.682017 20078 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:13:24.682544 20078 webserver.cc:460] Webserver started at http://127.31.227.1:34655/ using document root <none> and password file <none>
I20260812 00:13:24.682837 20078 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:24.682901 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:24.683027 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:13:24 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:24.701725 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "9892c36fb6ef4e398c871018bb793c23"
format_stamp: "Formatted at 2026-08-12 00:13:24 on dist-test-slave-zr1t"
server_key: "4122b68c590b1ac8280ff44618bfeb4f"
server_key_iv: "7c48a3788231d7ce683d3a5484cbf35e"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:24.702303 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "9892c36fb6ef4e398c871018bb793c23"
format_stamp: "Formatted at 2026-08-12 00:13:24 on dist-test-slave-zr1t"
server_key: "4122b68c590b1ac8280ff44618bfeb4f"
server_key_iv: "7c48a3788231d7ce683d3a5484cbf35e"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:24.703881 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:24.717330 20103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:24.717572 20078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:24.717672 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "9892c36fb6ef4e398c871018bb793c23"
format_stamp: "Formatted at 2026-08-12 00:13:24 on dist-test-slave-zr1t"
server_key: "4122b68c590b1ac8280ff44618bfeb4f"
server_key_iv: "7c48a3788231d7ce683d3a5484cbf35e"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:24.717785 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:24.729836 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:24.732059 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:24.732224 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:24.732474 20078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:24.732904 20078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:24.732986 20078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:24.733048 20078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:24.733094 20078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:24.741168 20078 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:37519
I20260812 00:13:24.741711 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:24.744029 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:37519 every 8 connection(s)
I20260812 00:13:24.744208 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 20078
I20260812 00:13:24.744338 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:24.744544 32652 external_mini_cluster.cc:1428] Setting key 6b089ca6732130e20225de6c3295c165
Aug 12 00:13:24 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
I20260812 00:13:24.758958 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:24.745132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:41999 (local address 127.31.227.62:35635)
0812 00:13:24.746105 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:24.746108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:24.746119 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:24.746628 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:24.746629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:24.746645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:24.746711 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:24.747508 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:24.748263 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:24.749920 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:24.750225 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:24.750282 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:24.753773 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:24.753775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:24.753775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:24.753783 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:24.755670 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:24.757554 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:24.757556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:24.757556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:24.757630 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:24.757967 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:24.757969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:24.757969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:24.758126 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:24.758212 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:24.758753 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 00:13:24.758850 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":88,"threads_started":1}
I20260812 00:13:24.759157 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:24.744733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:35635 (local address 127.31.227.1:41999)
0812 00:13:24.745120 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:24.745163 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:24.746447 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:24.746845 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:24.746850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:24.747102 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:24.747395 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:24.747402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:24.748380 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:24.748381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:24.749809 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:24.749812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:24.750385 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:24.750386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:24.750546 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:24.751412 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:24.751436 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:24.753653 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:24.757062 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:24.757071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:24.757073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:24.757423 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:24.757703 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:24.757704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:24.757704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:24.757854 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:24.758219 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:24.758226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:24.758558 (+ 332us) client_negotiation.cc:769] Sending connection context
0812 00:13:24.758678 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:13:24.758846 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":74,"threads_started":1}
I20260812 00:13:24.759905 20291 heartbeater.cc:347] Connected to a master server at 127.31.227.62:35635
I20260812 00:13:24.760054 20291 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
I20260812 00:13:24.760257 20291 heartbeater.cc:511] Master 127.31.227.62:35635 requested a full tablet report, sending...
I20260812 00:13:24.760874 19935 ts_manager.cc:194] Registered new tserver with Master: 9892c36fb6ef4e398c871018bb793c23 (127.31.227.1:37519)
I20260812 00:13:24.762109 19935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:41999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:13:24.777861 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:40947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35635
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:24.897092 20301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:24.897413 20301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:24.897508 20301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:24.899241 20301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:24.899344 20301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:24.899422 20301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:13:24.901510 20301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.227.20:40947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:13:24.902091 20301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:24.902415 20301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:24.909255 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:24.909677 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:24.909802 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:24.910210 20301 server_base.cc:1034] running on GCE node
I20260812 00:13:24.910526 20301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:24.910786 20301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:24.911954 20301 hybrid_clock.cc:648] HybridClock initialized: now 1786493604911919 us; error 48 us; skew 500 ppm
Aug 12 00:13:24 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:24.915338 20301 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:13:24.915828 20301 webserver.cc:460] Webserver started at http://127.31.227.2:43957/ using document root <none> and password file <none>
I20260812 00:13:24.916100 20301 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:24.916157 20301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:24.916281 20301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:13:24 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:24.936805 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "9f41cd3b3755465d92e2846d573a5dc6"
format_stamp: "Formatted at 2026-08-12 00:13:24 on dist-test-slave-zr1t"
server_key: "934a34c41923e25bbf863cb9e666fa40"
server_key_iv: "9f3437a0ba18cff4230a76c8e802c674"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:24.937330 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "9f41cd3b3755465d92e2846d573a5dc6"
format_stamp: "Formatted at 2026-08-12 00:13:24 on dist-test-slave-zr1t"
server_key: "934a34c41923e25bbf863cb9e666fa40"
server_key_iv: "9f3437a0ba18cff4230a76c8e802c674"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:24.940999 20301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260812 00:13:24.958642 20327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:24.958877 20301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 00:13:24.958984 20301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "9f41cd3b3755465d92e2846d573a5dc6"
format_stamp: "Formatted at 2026-08-12 00:13:24 on dist-test-slave-zr1t"
server_key: "934a34c41923e25bbf863cb9e666fa40"
server_key_iv: "9f3437a0ba18cff4230a76c8e802c674"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:24.959084 20301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 00:13:24 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:24.978025 20301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:24.980455 20301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:24.980669 20301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:24.980981 20301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:24.981431 20301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:24.981554 20301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:24.981634 20301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:24.981698 20301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:25.000680 20301 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:37443
I20260812 00:13:25.006577 20301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:25.006032 20519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:37443 every 8 connection(s)
I20260812 00:13:25.016981 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 20301
I20260812 00:13:25.017110 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:25.017321 32652 external_mini_cluster.cc:1428] Setting key b9601eee3309c87195ac1693cc4cd06a
Aug 12 00:13:25 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:13:25.023765 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.010110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:47785 (local address 127.31.227.62:35635)
0812 00:13:25.010176 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.010179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.010486 (+ 307us) server_negotiation.cc:407] Connection header received
0812 00:13:25.011216 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.011218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.011235 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.011269 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:25.012188 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.012974 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.014557 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.014870 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.014917 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.020208 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:25.020209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:25.020210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:25.020218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:25.022127 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:25.022677 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:25.022678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:25.022678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:25.022738 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:25.023020 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:25.023021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:25.023022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:25.023155 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:25.023196 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.023573 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.023666 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:13:25.024317 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.009832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:35635 (local address 127.31.227.2:47785)
0812 00:13:25.010216 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.010262 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.011082 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.011484 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.011489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.011757 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:25.012058 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.012068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.013101 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.013102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.014438 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.014452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.015027 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.015028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.015175 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.016368 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:25.016378 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:25.018714 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:25.022241 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:25.022249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:25.022251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:25.022583 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:25.022819 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:25.022819 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:25.022819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:25.022947 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:25.023241 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:25.023247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:25.023410 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.023910 (+ 500us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.024079 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":98,"threads_started":1}
I20260812 00:13:25.024901 20520 heartbeater.cc:347] Connected to a master server at 127.31.227.62:35635
I20260812 00:13:25.025003 20520 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:25.025213 20520 heartbeater.cc:511] Master 127.31.227.62:35635 requested a full tablet report, sending...
I20260812 00:13:25.025616 19935 ts_manager.cc:194] Registered new tserver with Master: 9f41cd3b3755465d92e2846d573a5dc6 (127.31.227.2:37443)
I20260812 00:13:25.026147 19935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:47785
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:13:25.053462 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:40947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35635
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:36741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:25.176589 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:25.176940 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:25.177035 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:25.178862 20531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:25.178972 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:25.179059 20531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:13:25.181344 20531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:36741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.227.20:40947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:35635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:13:25.181843 20531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:25.182168 20531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:25.189009 20543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:25.189399 20542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:25.189599 20531 server_base.cc:1034] running on GCE node
W20260812 00:13:25.189677 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:25.189953 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:25.190225 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:25.191395 20531 hybrid_clock.cc:648] HybridClock initialized: now 1786493605191374 us; error 33 us; skew 500 ppm
Aug 12 00:13:25 dist-test-slave-zr1t krb5kdc[19103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:25.194718 20531 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:13:25.195220 20531 webserver.cc:460] Webserver started at http://127.31.227.3:46575/ using document root <none> and password file <none>
I20260812 00:13:25.195469 20531 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:25.195518 20531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:25.195639 20531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:13:25 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:25.216404 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "4de3a234b82944498093a888e7d60056"
format_stamp: "Formatted at 2026-08-12 00:13:25 on dist-test-slave-zr1t"
server_key: "eb8bfb347ac80cad4c25102c2210156c"
server_key_iv: "66a0642fee164f886d1a105be4a290c5"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:25.217769 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "4de3a234b82944498093a888e7d60056"
format_stamp: "Formatted at 2026-08-12 00:13:25 on dist-test-slave-zr1t"
server_key: "eb8bfb347ac80cad4c25102c2210156c"
server_key_iv: "66a0642fee164f886d1a105be4a290c5"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:25.229640 20531 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.001s sys 0.001s
I20260812 00:13:25.247231 20556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:25.247503 20531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 00:13:25.247649 20531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "4de3a234b82944498093a888e7d60056"
format_stamp: "Formatted at 2026-08-12 00:13:25 on dist-test-slave-zr1t"
server_key: "eb8bfb347ac80cad4c25102c2210156c"
server_key_iv: "66a0642fee164f886d1a105be4a290c5"
server_key_version: "kuduclusterkey@0"
I20260812 00:13:25.247838 20531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:25.271595 20531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:25.274132 20531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:25.274359 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:25.274636 20531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:25.275053 20531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:25.275141 20531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:25.275220 20531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:25.275278 20531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:25.282042 20531 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:43165
I20260812 00:13:25.282624 20531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:13:25.285365 20733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:43165 every 8 connection(s)
I20260812 00:13:25.290719 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 20531
I20260812 00:13:25.290844 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:25.291041 32652 external_mini_cluster.cc:1428] Setting key c1a1d11e50e22687660f3a06083a3f46
Aug 12 00:13:25 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:13:25.299340 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.286120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:33471 (local address 127.31.227.62:35635)
0812 00:13:25.286196 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.286199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.289875 (+ 3676us) server_negotiation.cc:407] Connection header received
0812 00:13:25.290442 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.290443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.290459 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.290516 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:25.291972 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.292732 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.293869 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.294163 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.294242 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.296324 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:25.296326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:25.296326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:25.296334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:25.297863 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:25.298281 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:25.298282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:25.298283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:25.298348 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:25.298592 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:25.298593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:25.298593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:25.298737 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:25.298808 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.299127 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.299222 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:13:25.299525 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.286101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:35635 (local address 127.31.227.3:33471)
0812 00:13:25.289717 (+ 3616us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.289762 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.290320 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.291237 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.291243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.291529 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:25.291847 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.291854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.292859 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.292860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.293778 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.293781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.294251 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.294252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.294351 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.294951 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:25.294960 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:25.296226 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:25.297959 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:25.297965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:25.297966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:25.298184 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:25.298418 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:25.298418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:25.298418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:25.298525 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:25.298803 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:25.298808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:25.298924 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.299090 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.299278 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3564,"thread_start_us":91,"threads_started":1}
I20260812 00:13:25.300026 20734 heartbeater.cc:347] Connected to a master server at 127.31.227.62:35635
I20260812 00:13:25.300143 20734 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:25.300331 20734 heartbeater.cc:511] Master 127.31.227.62:35635 requested a full tablet report, sending...
I20260812 00:13:25.300761 19935 ts_manager.cc:194] Registered new tserver with Master: 4de3a234b82944498093a888e7d60056 (127.31.227.3:43165)
I20260812 00:13:25.301136 19935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:33471
I20260812 00:13:25.302678 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:25.311957 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.305638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.31.227.62:35635)
0812 00:13:25.305678 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.305680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.305810 (+ 130us) server_negotiation.cc:407] Connection header received
0812 00:13:25.305888 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.305889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.305899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.305962 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:25.306531 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.307009 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.307758 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.307958 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.308034 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.309257 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:25.309259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:25.309260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:25.309268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:25.310807 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:25.311173 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:25.311174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:25.311174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:25.311234 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:25.311449 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:25.311450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:25.311450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:25.311559 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:25.311664 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.311774 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.311854 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:13:25.313946 19935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:25.315606 19935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:25.331871 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.325806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:37443 (local address 127.0.0.1:45398)
0812 00:13:25.327285 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.327329 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.327407 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.327723 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.327726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.327728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:25.327796 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.327798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.329125 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.329126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.330500 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.330503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.331573 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.331574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.331657 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.331669 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.331736 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.331786 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"thread_start_us":112,"threads_started":1}
I20260812 00:13:25.332463 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.326379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.31.227.2:37443)
0812 00:13:25.326591 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.326596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.327506 (+ 910us) server_negotiation.cc:407] Connection header received
0812 00:13:25.327552 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.327555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.327570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.327638 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:25.327874 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.329026 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.330623 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.331382 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.331464 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.331536 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.332324 (+ 788us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.332367 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":118,"threads_started":1}
I20260812 00:13:25.333285 20406 tablet_service.cc:1467] Processing CreateTablet for tablet 6a612b311a9b47ecb7b990f4bdf04896 (DEFAULT_TABLE table=test-table [id=8b15fe2275c34880834223330ac14a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:25.333722 20406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a612b311a9b47ecb7b990f4bdf04896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:25.337661 20773 tablet_bootstrap.cc:492] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: Bootstrap starting.
I20260812 00:13:25.338799 20773 tablet_bootstrap.cc:654] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:25.339486 20773 log.cc:824] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:25.340293 20773 tablet_bootstrap.cc:492] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: No bootstrap required, opened a new log
I20260812 00:13:25.340359 20773 ts_tablet_manager.cc:1397] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:13:25.342259 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.326182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.31.227.1:37519)
0812 00:13:25.328373 (+ 2191us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.328378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.328710 (+ 332us) server_negotiation.cc:407] Connection header received
0812 00:13:25.328824 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.328827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.328843 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.328895 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:25.330913 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.333548 (+ 2635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.341005 (+ 7457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.341772 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.341825 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.341883 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.342136 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.342182 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":78,"threads_started":1}
I20260812 00:13:25.342361 20773 raft_consensus.cc:348] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.342523 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.326071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:37519 (local address 127.0.0.1:37704)
0812 00:13:25.328650 (+ 2579us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.328660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.328737 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.329013 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.329014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.329016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:25.329061 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.329063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.337462 (+ 8399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.337463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.338665 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.338669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.341961 (+ 3292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.341962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.342031 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.342035 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.342404 (+ 369us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.342453 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2564,"thread_start_us":65,"threads_started":1}
I20260812 00:13:25.342532 20773 raft_consensus.cc:374] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:25.342567 20773 raft_consensus.cc:729] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f41cd3b3755465d92e2846d573a5dc6, State: Initialized, Role: FOLLOWER
I20260812 00:13:25.342679 20773 consensus_queue.cc:260] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.342929 20773 ts_tablet_manager.cc:1428] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:25.343237 20166 tablet_service.cc:1467] Processing CreateTablet for tablet 6a612b311a9b47ecb7b990f4bdf04896 (DEFAULT_TABLE table=test-table [id=8b15fe2275c34880834223330ac14a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:25.343607 20166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a612b311a9b47ecb7b990f4bdf04896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:25.343796 20520 heartbeater.cc:503] Master 127.31.227.62:35635 was elected leader, sending a full tablet report...
I20260812 00:13:25.347213 20778 tablet_bootstrap.cc:492] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: Bootstrap starting.
I20260812 00:13:25.347700 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.325807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43165 (local address 127.0.0.1:35208)
0812 00:13:25.328185 (+ 2378us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.328197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.328256 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.329155 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.329156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.329159 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:25.329554 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.329557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.338799 (+ 9242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.338800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.339921 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.339924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.347427 (+ 7503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.347428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.347504 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.347509 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.347568 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.347615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2357,"thread_start_us":114,"threads_started":1}
I20260812 00:13:25.347719 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.326306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.31.227.3:43165)
0812 00:13:25.328464 (+ 2158us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.328468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.328477 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:13:25.328517 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.328520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.328536 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.329649 (+ 1113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:25.329727 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.330790 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.340074 (+ 9284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.340874 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.347426 (+ 6552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.347544 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.347581 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.347633 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2121,"thread_start_us":89,"threads_started":1}
I20260812 00:13:25.348372 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 6a612b311a9b47ecb7b990f4bdf04896 (DEFAULT_TABLE table=test-table [id=8b15fe2275c34880834223330ac14a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:25.348738 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a612b311a9b47ecb7b990f4bdf04896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:25.348856 20778 tablet_bootstrap.cc:654] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:25.349231 20778 log.cc:824] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:25.352713 20780 tablet_bootstrap.cc:492] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Bootstrap starting.
I20260812 00:13:25.353868 20780 tablet_bootstrap.cc:654] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:25.354221 20780 log.cc:824] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:25.354784 20778 tablet_bootstrap.cc:492] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: No bootstrap required, opened a new log
I20260812 00:13:25.354866 20778 ts_tablet_manager.cc:1397] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260812 00:13:25.357573 20778 raft_consensus.cc:348] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.357796 20778 raft_consensus.cc:374] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:25.357872 20778 raft_consensus.cc:729] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9892c36fb6ef4e398c871018bb793c23, State: Initialized, Role: FOLLOWER
I20260812 00:13:25.358239 20780 tablet_bootstrap.cc:492] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: No bootstrap required, opened a new log
I20260812 00:13:25.358372 20780 ts_tablet_manager.cc:1397] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 00:13:25.358639 20778 consensus_queue.cc:260] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.358987 20778 ts_tablet_manager.cc:1428] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 00:13:25.359170 20291 heartbeater.cc:503] Master 127.31.227.62:35635 was elected leader, sending a full tablet report...
I20260812 00:13:25.360630 20780 raft_consensus.cc:348] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.360795 20780 raft_consensus.cc:374] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:25.360872 20780 raft_consensus.cc:729] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de3a234b82944498093a888e7d60056, State: Initialized, Role: FOLLOWER
I20260812 00:13:25.361019 20780 consensus_queue.cc:260] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.361272 20780 ts_tablet_manager.cc:1428] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:25.361413 20734 heartbeater.cc:503] Master 127.31.227.62:35635 was elected leader, sending a full tablet report...
Aug 12 00:13:25 dist-test-slave-zr1t krb5kdc[19103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493604, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:13:25.473336 20776 raft_consensus.cc:482] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:25.473484 20776 raft_consensus.cc:504] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.473904 20776 leader_election.cc:288] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4de3a234b82944498093a888e7d60056 (127.31.227.3:43165), 9892c36fb6ef4e398c871018bb793c23 (127.31.227.1:37519)
I20260812 00:13:25.478963 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.474458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:57947 (local address 127.31.227.3:43165)
0812 00:13:25.474505 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.474507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.475081 (+ 574us) server_negotiation.cc:407] Connection header received
0812 00:13:25.475179 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.475180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.475191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.475220 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:25.475483 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.476258 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.478019 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.478584 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.478630 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.478639 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.478851 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.478889 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:13:25.479382 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.474247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43165 (local address 127.31.227.2:57947)
0812 00:13:25.475027 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.475035 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.475105 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.475288 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.475289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.475290 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:25.475376 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.475378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.476657 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.476658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.477911 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.477913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.478699 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.478700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.478773 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.478776 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.479249 (+ 473us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.479296 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":16,"thread_start_us":57,"threads_started":1}
I20260812 00:13:25.479849 20668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a612b311a9b47ecb7b990f4bdf04896" candidate_uuid: "9f41cd3b3755465d92e2846d573a5dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de3a234b82944498093a888e7d60056" is_pre_election: true
I20260812 00:13:25.480003 20668 raft_consensus.cc:2454] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f41cd3b3755465d92e2846d573a5dc6 in term 0.
I20260812 00:13:25.480307 20329 leader_election.cc:302] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4de3a234b82944498093a888e7d60056, 9f41cd3b3755465d92e2846d573a5dc6; no voters:
I20260812 00:13:25.480433 20776 raft_consensus.cc:2781] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:25.480482 20776 raft_consensus.cc:482] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:25.480502 20776 raft_consensus.cc:3037] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:25.481359 20776 raft_consensus.cc:504] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.481495 20776 leader_election.cc:288] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [CANDIDATE]: Term 1 election: Requested vote from peers 4de3a234b82944498093a888e7d60056 (127.31.227.3:43165), 9892c36fb6ef4e398c871018bb793c23 (127.31.227.1:37519)
I20260812 00:13:25.481835 20668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a612b311a9b47ecb7b990f4bdf04896" candidate_uuid: "9f41cd3b3755465d92e2846d573a5dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de3a234b82944498093a888e7d60056"
I20260812 00:13:25.481909 20668 raft_consensus.cc:3037] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:25.482738 20668 raft_consensus.cc:2454] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f41cd3b3755465d92e2846d573a5dc6 in term 1.
I20260812 00:13:25.482960 20329 leader_election.cc:302] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4de3a234b82944498093a888e7d60056, 9f41cd3b3755465d92e2846d573a5dc6; no voters:
I20260812 00:13:25.483078 20776 raft_consensus.cc:2781] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:25.483237 20776 raft_consensus.cc:686] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [term 1 LEADER]: Becoming Leader. State: Replica: 9f41cd3b3755465d92e2846d573a5dc6, State: Running, Role: LEADER
I20260812 00:13:25.483321 20776 consensus_queue.cc:237] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } }
I20260812 00:13:25.484140 19935 catalog_manager.cc:5495] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f41cd3b3755465d92e2846d573a5dc6 (127.31.227.2). New cstate: current_term: 1 leader_uuid: "9f41cd3b3755465d92e2846d573a5dc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f41cd3b3755465d92e2846d573a5dc6" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 37443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:25.484967 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.474139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:37519 (local address 127.31.227.2:45505)
0812 00:13:25.474765 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.474777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.474826 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.475444 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:25.475445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:25.475449 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:25.475517 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.475519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.476396 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.476397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.483909 (+ 7512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:25.483913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.484700 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.484701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:25.484782 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.484786 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:25.484837 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:13:25.484887 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605}
I20260812 00:13:25.485037 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.474970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:45505 (local address 127.31.227.1:37519)
0812 00:13:25.475266 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.475268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.475279 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:25.475309 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.475310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.475321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.475386 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:25.475601 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.476307 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.484030 (+ 7723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.484609 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.484702 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.484712 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.484922 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.484971 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260812 00:13:25.485641 20195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a612b311a9b47ecb7b990f4bdf04896" candidate_uuid: "9f41cd3b3755465d92e2846d573a5dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9892c36fb6ef4e398c871018bb793c23" is_pre_election: true
I20260812 00:13:25.485809 20195 raft_consensus.cc:2454] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f41cd3b3755465d92e2846d573a5dc6 in term 0.
I20260812 00:13:25.485950 20196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a612b311a9b47ecb7b990f4bdf04896" candidate_uuid: "9f41cd3b3755465d92e2846d573a5dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9892c36fb6ef4e398c871018bb793c23"
I20260812 00:13:25.486020 20196 raft_consensus.cc:3037] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:25.486672 20196 raft_consensus.cc:2454] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f41cd3b3755465d92e2846d573a5dc6 in term 1.
W20260812 00:13:25.494817 20292 tablet.cc:2369] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:25.495997 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.492638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.31.227.2:37443)
0812 00:13:25.492680 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:25.492682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:25.492691 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:13:25.492759 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:25.492760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:25.492779 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:25.492853 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:25.493056 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.493661 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.495105 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:25.495378 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:25.495446 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:25.495649 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:25.495758 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:25.495886 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:25.495907 (+ 21us) server_negotiation.cc:299] Negotiation successful
0812 00:13:25.495923 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:13:25.497987 20196 raft_consensus.cc:1261] T 6a612b311a9b47ecb7b990f4bdf04896 P 9892c36fb6ef4e398c871018bb793c23 [term 1 FOLLOWER]: Refusing update from remote peer 9f41cd3b3755465d92e2846d573a5dc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:25.498031 20668 raft_consensus.cc:1261] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 1 FOLLOWER]: Refusing update from remote peer 9f41cd3b3755465d92e2846d573a5dc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:25.498382 20792 consensus_queue.cc:1035] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4de3a234b82944498093a888e7d60056" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:25.498519 20776 consensus_queue.cc:1035] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9892c36fb6ef4e398c871018bb793c23" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:25.503599 20799 mvcc.cc:205] Tried to move back new op lower bound from 7317477808116744192 to 7317477808060289024. Current Snapshot: MvccSnapshot[applied={T|T < 7317477808116744192}]
I20260812 00:13:25.504077 20798 mvcc.cc:205] Tried to move back new op lower bound from 7317477808116744192 to 7317477808060289024. Current Snapshot: MvccSnapshot[applied={T|T < 7317477808116744192}]
I20260812 00:13:25.507851 19932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:25.507954 19932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:25.508785 19932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c85e734ab3b4085a1f22d611eaf5d3c: Sending DeleteTablet for 3 replicas of tablet 6a612b311a9b47ecb7b990f4bdf04896
I20260812 00:13:25.509230 20634 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a612b311a9b47ecb7b990f4bdf04896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:25 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:35208
I20260812 00:13:25.509310 20166 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a612b311a9b47ecb7b990f4bdf04896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:25 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:37704
I20260812 00:13:25.509713 20406 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a612b311a9b47ecb7b990f4bdf04896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:25 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45398
I20260812 00:13:25.509758 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 20078
I20260812 00:13:25.527132 20810 tablet_replica.cc:330] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: stopping tablet replica
I20260812 00:13:25.527351 20810 raft_consensus.cc:2229] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:25.527590 20810 raft_consensus.cc:2258] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:25.528395 20810 ts_tablet_manager.cc:1905] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:25.530270 20810 ts_tablet_manager.cc:1918] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:25.533605 20810 log.cc:1196] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/6a612b311a9b47ecb7b990f4bdf04896
I20260812 00:13:25.534351 20810 ts_tablet_manager.cc:1939] T 6a612b311a9b47ecb7b990f4bdf04896 P 4de3a234b82944498093a888e7d60056: Deleting consensus metadata
I20260812 00:13:25.534910 19912 catalog_manager.cc:4841] TS 4de3a234b82944498093a888e7d60056 (127.31.227.3:43165): tablet 6a612b311a9b47ecb7b990f4bdf04896 (table test-table [id=8b15fe2275c34880834223330ac14a45]) successfully deleted
W20260812 00:13:25.537281 19913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:37519 (error 108)
I20260812 00:13:25.538030 20812 tablet_replica.cc:330] T 6a612b311a9b47ecb7b990f4bdf04896 P 9f41cd3b3755465d92e2846d573a5dc6: stopping tablet replica
I20260812 00:13:25.538245 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 20301
I20260812 00:13:25.553539 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:25.553368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:37519 (local address 127.0.0.1:37710)
0812 00:13:25.553429 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.553478 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:13:25.558332 19913 catalog_manager.cc:4584] TS 9892c36fb6ef4e398c871018bb793c23 (127.31.227.1:37519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a612b311a9b47ecb7b990f4bdf04896: Network error: Client connection negotiation failed: client connection to 127.31.227.1:37519: connect: Connection refused (error 111)
W20260812 00:13:25.563643 20762 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:13:25.563352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:37443 (local address 127.0.0.1:45422)
0812 00:13:25.563412 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:25.563424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:25.563477 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:25.563589 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.2:37443: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:13:25.564160 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 20531
I20260812 00:13:25.583046 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 19876
I20260812 00:13:25.604569 32652 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 00:13:25.949174 32652 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 00:13:25.949275 32652 mini_ranger.cc:66] Stopping Ranger...
I20260812 00:13:26.304106 32652 mini_ranger.cc:68] Stopped Ranger
2026-08-12 00:13:26.304 UTC [19159] LOG: received smart shutdown request
2026-08-12 00:13:26.308 UTC [19159] LOG: background worker "logical replication launcher" (PID 19176) exited with exit code 1
2026-08-12 00:13:26.308 UTC [19166] LOG: shutting down
2026-08-12 00:13:26.457 UTC [19159] LOG: database system is shut down
2026-08-12T00:13:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48043 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:13:26 dist-test-slave-zr1t krb5kdc[20830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:13:26 dist-test-slave-zr1t krb5kdc[20830](info): set up 2 sockets
Aug 12 00:13:26 dist-test-slave-zr1t krb5kdc[20830](info): commencing operation
krb5kdc: starting...
W20260812 00:13:27.877205 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:13:27 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/rangeradmin_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/rangeradmin_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/rangerlookup_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/rangerlookup_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
I20260812 00:13:28.048717 32652 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSkgTkG/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/postgres -l logfile start
2026-08-12 00:13:30.348 UTC [20882] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 00:13:30.348 UTC [20882] LOG: listening on IPv4 address "127.31.227.20", port 45703
2026-08-12 00:13:30.351 UTC [20882] LOG: listening on Unix socket "/tmp/.s.PGSQL.45703"
2026-08-12 00:13:30.372 UTC [20885] LOG: database system was shut down at 2026-08-12 00:13:29 UTC
2026-08-12 00:13:30.377 UTC [20882] LOG: database system is ready to accept connections
I20260812 00:13:31.585016 32652 mini_postgres.cc:96] Postgres bound to 45703
2026-08-12 00:13:31.592 UTC [20904] FATAL: database "slave" does not exist
127.31.227.20:45703 - accepting connections
I20260812 00:13:31.592921 32652 mini_ranger.cc:161] Starting Ranger...
I20260812 00:13:31.608409 32652 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 00:13:32.032189 32652 mini_ranger.cc:87] Created ranger Postgres database
I20260812 00:13:32.032280 32652 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-admin
2026-08-12 00:13:32,337 [I] DB FLAVOR :POSTGRES
2026-08-12 00:13:32,338 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:13:32,338 [I] Checking connection..
2026-08-12 00:13:32,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 00:13:32,581 [I] Checking connection passed.
2026-08-12 00:13:32,582 [I] --------- Verifying version history table ---------
2026-08-12 00:13:32,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:13:32,853 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:13:32,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:13:33,121 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:13:33,121 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 00:13:33,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 00:13:33.382 UTC [20999] WARNING: there is no transaction in progress
2026-08-12 00:13:33,394 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 00:13:33,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:13:33,648 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 00:13:33,648 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:13:33,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 00:13:33,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 00:13:34,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zr1t.c.gcp-upstream.internal','N') ;"
2026-08-12 00:13:34,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:13:34,693 [I] Table x_portal_user does not exist in database ranger
2026-08-12 00:13:34,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:13:34,964 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 00:13:34,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:13:35,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:13:35,460 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 00:13:35,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 00:13:36.392 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.408 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.425 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.440 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.455 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.472 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.525 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.532 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.545 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.554 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.563 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.576 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.585 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.591 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.598 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:36.604 UTC [21192] WARNING: there is no transaction in progress
2026-08-12 00:13:37,214 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 00:13:37,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 00:13:37,462 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 00:13:37,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:13:37,720 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 00:13:37,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:13:37,985 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 00:13:37,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:13:38,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:13:38,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zr1t.c.gcp-upstream.internal';"
2026-08-12 00:13:38,717 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 00:13:38,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 00:13:38,985 [I] DB_PATCHES have already been applied
I20260812 00:13:38.991940 32652 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSkgTkG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 00:13:38.992039 32652 mini_ranger.cc:193] Using host: 127.31.227.20
I20260812 00:13:38.996572 32652 mini_ranger.cc:227] Ranger admin URL: http://127.31.227.20:34589
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 00:13:40.413448 32652 mini_ranger.cc:160] Time spent starting Ranger: real 8.821s user 0.000s sys 0.012s
Aug 12, 2026 12:14:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 12:14:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 12:14:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 12:14:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 12:14:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 00:14:02.485667 32652 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.227.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.227.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/rangerkms_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.227.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/rangerkms_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.227.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.227.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.227.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/HTTP_127.31.227.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 00:14:02 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493642, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 00:14:02.568073 32652 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 00:14:02.577679 32652 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 00:14:03.341429 32652 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 00:14:03.341527 32652 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms
I20260812 00:14:03.341572 32652 mini_ranger_kms.cc:227] Using postgres at 127.31.227.20:45703
2026-08-12 00:14:03,533 [I] DB FLAVOR :POSTGRES
2026-08-12 00:14:03,533 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:14:03,533 [I] Checking connection
2026-08-12 00:14:03,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 00:14:03,780 [I] connection success
2026-08-12 00:14:03,780 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 00:14:03,780 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 00:14:03,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 00:14:04,037 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 00:14:04,037 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:14:04,037 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 00:14:04,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.227.20:45703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 00:14:04,298 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 00:14:04.780087 32652 mini_ranger_kms.cc:310] Created kms service
I20260812 00:14:04.988432 32652 mini_ranger_kms.cc:326] Created kudu user
I20260812 00:14:05.033895 32652 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 00:14:05 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493642, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:05.450491 32652 mini_ranger_kms.cc:384] Added ranger policy
I20260812 00:14:05.450730 32652 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSkgTkG/build/release/bin/postgresql.jar:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSkgTkG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 00:14:05.450755 32652 mini_ranger_kms.cc:240] Using host: 127.31.227.20
I20260812 00:14:05.455812 32652 mini_ranger_kms.cc:276] Ranger KMS PID: 21507
I20260812 00:14:05.455906 32652 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.227.20:52521
Aug 12, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 00:14:07 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493647, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.227.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 00:14:08 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493647, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.227.20@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:08.338315 32652 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.770s user 0.010s sys 0.165s
I20260812 00:14:08.338424 32652 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 00:14:08.338469 32652 mini_ranger_kms.cc:401] 127.31.227.20:52521/kms/v1/keys
Aug 12 00:14:08 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493647, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.227.20@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
Aug 12 00:14:10 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:14:10.946635 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38005
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:38005
--ranger_config_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:11.031597 21603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:11.031809 21603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:11.031831 21603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:11.033039 21603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:11.033084 21603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:11.033098 21603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:11.033111 21603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:11.033123 21603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:11.034731 21603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.227.20:52521/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:38005
--ranger_config_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:11.035000 21603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:11.035271 21603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:11.040620 21611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:11.040741 21613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:11.040627 21616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:11.040644 21603 server_base.cc:1034] running on GCE node
I20260812 00:14:11.041152 21603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:11.041505 21603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:11.042729 21603 hybrid_clock.cc:648] HybridClock initialized: now 1786493651042700 us; error 43 us; skew 500 ppm
Aug 12 00:14:11 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:11.046790 21603 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:11.047397 21603 webserver.cc:460] Webserver started at http://127.31.227.62:35125/ using document root <none> and password file <none>
I20260812 00:14:11.047778 21603 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:11.047842 21603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:11.048027 21603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:14:11 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:11.106388 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "632cab685d244b52890f17e30b2714fe"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zr1t"
server_key: "83de9ad7a917a9893ef13bae3b666dfb"
server_key_iv: "c4a9c60a91536c7fa648997e5bbc8d91"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:11.106941 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "632cab685d244b52890f17e30b2714fe"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zr1t"
server_key: "83de9ad7a917a9893ef13bae3b666dfb"
server_key_iv: "c4a9c60a91536c7fa648997e5bbc8d91"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:11.108765 21603 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 00:14:11.123838 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:11.124053 21603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:11.124152 21603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "632cab685d244b52890f17e30b2714fe"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zr1t"
server_key: "83de9ad7a917a9893ef13bae3b666dfb"
server_key_iv: "c4a9c60a91536c7fa648997e5bbc8d91"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:11.124256 21603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:11.156533 21603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:11.158788 21603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:11.158960 21603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:11.164606 21603 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:38005
I20260812 00:14:11.164639 21734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:38005 every 8 connection(s)
I20260812 00:14:11.165089 21603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:11.165897 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:11.168460 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Bootstrap starting.
I20260812 00:14:11.169070 21738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:11.169390 21738 log.cc:824] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:11.170159 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: No bootstrap required, opened a new log
I20260812 00:14:11.172189 21738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:11.172390 21738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:11.172463 21738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632cab685d244b52890f17e30b2714fe, State: Initialized, Role: FOLLOWER
I20260812 00:14:11.172647 21738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:11.172749 21738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:11.172792 21738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:11.172839 21738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:11.173380 21738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:11.173502 21738 leader_election.cc:302] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 632cab685d244b52890f17e30b2714fe; no voters:
I20260812 00:14:11.173720 21738 leader_election.cc:288] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:11.173785 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 21603
I20260812 00:14:11.173780 21741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:11.173908 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:14:11.173974 21741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 LEADER]: Becoming Leader. State: Replica: 632cab685d244b52890f17e30b2714fe, State: Running, Role: LEADER
I20260812 00:14:11.174059 21741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:11.174099 21738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:11.174129 32652 external_mini_cluster.cc:1428] Setting key a9f4b0fd833d83a314db1184114c47d1
I20260812 00:14:11.174355 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632cab685d244b52890f17e30b2714fe. Latest consensus state: current_term: 1 leader_uuid: "632cab685d244b52890f17e30b2714fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } } }
I20260812 00:14:11.174417 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:11.174530 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "632cab685d244b52890f17e30b2714fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } } }
I20260812 00:14:11.174567 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:11.174664 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:11.175148 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:11.175709 21738 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 00:14:11.176962 21776 mvcc.cc:205] Tried to move back new op lower bound from 7317477995218952192 to 7317477995209191424. Current Snapshot: MvccSnapshot[applied={T|T < 7317477995218952192}]
I20260812 00:14:11.178449 21750 catalog_manager.cc:1347] Generated new cluster ID: c1d642ce1f1a48df9c2cc78df19128f8
I20260812 00:14:11.178576 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:14:11 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:11.189962 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:11.174775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.31.227.62:38005)
0812 00:14:11.175131 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:11.175142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:11.175445 (+ 303us) server_negotiation.cc:407] Connection header received
0812 00:14:11.175786 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:11.175796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:11.175861 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:11.176023 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:11.176853 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:11.177976 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:11.180267 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:11.181881 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:11.182022 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:11.185193 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:11.185198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:11.185198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:11.185210 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:11.187495 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:11.188137 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:11.188140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:11.188140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:11.188231 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:11.188446 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:11.188447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:11.188447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:11.188752 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:11.188886 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:11.189505 (+ 619us) server_negotiation.cc:299] Negotiation successful
0812 00:14:11.189692 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":77,"threads_started":1}
I20260812 00:14:11.201771 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:11.202615 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:11.207849 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Generated new TSK 0
I20260812 00:14:11.208081 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:14:11.952776 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38005
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:12.043720 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:12.043924 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:12.043947 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:12.045104 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:12.045161 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:12.045205 21816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:14:12.046803 21816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.227.20:52521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:12.047062 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:12.047341 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:12.052688 21825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:12.052762 21827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:12.053136 21816 server_base.cc:1034] running on GCE node
W20260812 00:14:12.052842 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:12.053387 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:12.053689 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:12.054870 21816 hybrid_clock.cc:648] HybridClock initialized: now 1786493652054844 us; error 38 us; skew 500 ppm
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:12.061904 21816 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:14:12.062480 21816 webserver.cc:460] Webserver started at http://127.31.227.1:44103/ using document root <none> and password file <none>
I20260812 00:14:12.062727 21816 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:12.062795 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:12.062937 21816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:12.076082 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "93158a7ec913495c9e506ff62c4a8565"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "c837c6620fe1e38b1f3c9df9cceaf37a"
server_key_iv: "9ac81205ff2bd164ff97e16960d3d370"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.076562 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "93158a7ec913495c9e506ff62c4a8565"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "c837c6620fe1e38b1f3c9df9cceaf37a"
server_key_iv: "9ac81205ff2bd164ff97e16960d3d370"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.079577 21816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260812 00:14:12.096207 21840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.096436 21816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 00:14:12.096531 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "93158a7ec913495c9e506ff62c4a8565"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "c837c6620fe1e38b1f3c9df9cceaf37a"
server_key_iv: "9ac81205ff2bd164ff97e16960d3d370"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.096618 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:12.130199 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:12.132922 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:12.133133 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:12.133466 21816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:12.133922 21816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:12.134014 21816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.134114 21816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:12.134186 21816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.141834 21816 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:35709
I20260812 00:14:12.141958 22022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:35709 every 8 connection(s)
I20260812 00:14:12.142318 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:12.150280 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 21816
I20260812 00:14:12.150403 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:14:12.150633 32652 external_mini_cluster.cc:1428] Setting key e21dec4825cbc9a13516b7d3e6c0d950
I20260812 00:14:12.157646 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.143094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38005 (local address 127.31.227.1:56193)
0812 00:14:12.143287 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:12.143332 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:12.143825 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:12.144201 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:12.144206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:12.144425 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:12.144624 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:12.144628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.145433 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.145435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:12.146302 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:12.146305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.146744 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.146745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:12.146875 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.147478 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:12.147488 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:12.148920 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:12.155835 (+ 6915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:12.155846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:12.155850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:12.156196 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:12.156461 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:12.156462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:12.156462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:12.156601 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:12.156905 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:12.156912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:12.157090 (+ 178us) client_negotiation.cc:769] Sending connection context
0812 00:14:12.157185 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 00:14:12.157369 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260812 00:14:12.158478 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.143147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:56193 (local address 127.31.227.62:38005)
0812 00:14:12.143978 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:12.143980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:12.143987 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:14:12.144011 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:12.144012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:12.144024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:12.144115 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:12.144730 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.145334 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.146406 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.146628 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.146716 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.149026 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:12.149027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:12.149028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:12.149035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:12.155709 (+ 6674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.156297 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.156297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.156298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.156384 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.156679 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.156680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.156680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.156822 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:12.158201 (+ 1379us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:12.158248 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 00:14:12.158380 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":106,"threads_started":1}
I20260812 00:14:12.159037 22023 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38005
I20260812 00:14:12.159137 22023 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:12.159340 22023 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:12.159941 21665 ts_manager.cc:194] Registered new tserver with Master: 93158a7ec913495c9e506ff62c4a8565 (127.31.227.1:35709)
I20260812 00:14:12.160816 21665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:56193
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:14:12.183933 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38005
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:12.307590 22029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:12.307850 22029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:12.307891 22029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:12.309082 22029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:12.309155 22029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:12.309202 22029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:14:12.310757 22029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.227.20:52521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:12.311036 22029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:12.311306 22029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 00:14:12.316692 22051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:12.316731 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:12.317097 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:12.317564 22029 server_base.cc:1034] running on GCE node
I20260812 00:14:12.317791 22029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:12.318068 22029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:12.319224 22029 hybrid_clock.cc:648] HybridClock initialized: now 1786493652319200 us; error 39 us; skew 500 ppm
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:12.322256 22029 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:14:12.322780 22029 webserver.cc:460] Webserver started at http://127.31.227.2:36285/ using document root <none> and password file <none>
I20260812 00:14:12.323064 22029 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:12.323122 22029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:12.323256 22029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:12.343277 22029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "bcbdf5046a704a10875530956ed6c9cf"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3ea24edd4df9cfeda970dcf2ed23dc15"
server_key_iv: "fad1554824a39d8e45ff6697411cebcb"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.343765 22029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "bcbdf5046a704a10875530956ed6c9cf"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3ea24edd4df9cfeda970dcf2ed23dc15"
server_key_iv: "fad1554824a39d8e45ff6697411cebcb"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.349372 22029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 00:14:12.366065 22066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.366351 22029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:12.366477 22029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "bcbdf5046a704a10875530956ed6c9cf"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3ea24edd4df9cfeda970dcf2ed23dc15"
server_key_iv: "fad1554824a39d8e45ff6697411cebcb"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.366564 22029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:12.383281 22029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:12.385869 22029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:12.386072 22029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:12.386408 22029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:12.386862 22029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:12.386958 22029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.387041 22029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:12.387106 22029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.395097 22029 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:43997
I20260812 00:14:12.395145 22269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:43997 every 8 connection(s)
I20260812 00:14:12.395627 22029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:14:12.403208 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 22029
I20260812 00:14:12.403326 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:14:12.403585 32652 external_mini_cluster.cc:1428] Setting key 148864f767d3e5c7835af6d8c709f63f
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:12.418128 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.397260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:38549 (local address 127.31.227.62:38005)
0812 00:14:12.397308 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:12.397311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:12.397589 (+ 278us) server_negotiation.cc:407] Connection header received
0812 00:14:12.398629 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:12.398630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:12.398649 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:12.398731 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:12.399485 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.400295 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.401970 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.402305 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.402355 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.408543 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:12.408545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:12.408546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:12.408639 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:12.411196 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.412039 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.412041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.412041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.416830 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.417229 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.417230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.417230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.417403 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:12.417459 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:12.417906 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 00:14:12.418025 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:14:12.418987 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.396953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38005 (local address 127.31.227.2:38549)
0812 00:14:12.397427 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:12.397471 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:12.398479 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:12.398824 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:12.398829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:12.399084 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:12.399363 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:12.399369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.400420 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.400422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:12.401856 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:12.401860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.402481 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.402481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:12.402629 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.403957 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:12.403968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:12.408371 (+ 4403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:12.411319 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:12.411354 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:12.411358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:12.411871 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:12.416942 (+ 5071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:12.416943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:12.416944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:12.417139 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:12.417495 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:12.417502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:12.417728 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 00:14:12.418532 (+ 804us) client_negotiation.cc:240] Negotiation successful
0812 00:14:12.418734 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":101,"threads_started":1}
I20260812 00:14:12.419797 22270 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38005
I20260812 00:14:12.420048 22270 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:12.420284 22270 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:12.420730 21665 ts_manager.cc:194] Registered new tserver with Master: bcbdf5046a704a10875530956ed6c9cf (127.31.227.2:43997)
I20260812 00:14:12.421232 21665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:38549
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:14:12.451757 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38005
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:12.578258 22278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:12.578562 22278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:12.578603 22278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:12.580415 22278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:12.580487 22278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:12.580538 22278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:14:12.582883 22278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.227.20:52521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:12.583151 22278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:12.583412 22278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:12.588385 22285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:12.588409 22286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:12.588636 22289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:12.589020 22278 server_base.cc:1034] running on GCE node
I20260812 00:14:12.589242 22278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:12.589504 22278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:12.590703 22278 hybrid_clock.cc:648] HybridClock initialized: now 1786493652590685 us; error 33 us; skew 500 ppm
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:12.593613 22278 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:14:12.594201 22278 webserver.cc:460] Webserver started at http://127.31.227.3:36423/ using document root <none> and password file <none>
I20260812 00:14:12.594460 22278 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:12.594521 22278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:12.594661 22278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:12.608234 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "5d5ded3d07494d84a1b0bc1e90b233b7"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3a812612f839ac0c34aca4e327eea13e"
server_key_iv: "a7ae1afadd07d14ae9a6d869a720b372"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.608834 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "5d5ded3d07494d84a1b0bc1e90b233b7"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3a812612f839ac0c34aca4e327eea13e"
server_key_iv: "a7ae1afadd07d14ae9a6d869a720b372"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.613960 22278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260812 00:14:12.641700 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.641985 22278 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260812 00:14:12.642086 22278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "5d5ded3d07494d84a1b0bc1e90b233b7"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3a812612f839ac0c34aca4e327eea13e"
server_key_iv: "a7ae1afadd07d14ae9a6d869a720b372"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:12.642189 22278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:12.682343 22278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:12.685439 22278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:12.685706 22278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:12.685992 22278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:12.686431 22278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:12.686519 22278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.686592 22278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:12.686657 22278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.693668 22278 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41137
I20260812 00:14:12.694187 22278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:14:12.703169 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 22278
I20260812 00:14:12.703284 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:14:12.703503 32652 external_mini_cluster.cc:1428] Setting key 10ab0c38d21386261e868ec90dc48b14
I20260812 00:14:12.706701 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41137 every 8 connection(s)
Aug 12 00:14:12 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:12.718901 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.707738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:58901 (local address 127.31.227.62:38005)
0812 00:14:12.707785 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:12.707787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:12.708001 (+ 214us) server_negotiation.cc:407] Connection header received
0812 00:14:12.708758 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:12.708759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:12.708773 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:12.708806 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:12.709742 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.710517 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.712091 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.712391 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.712437 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.715337 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:12.715339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:12.715339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:12.715347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:12.717145 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.717720 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.717722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.717722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.717786 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.718104 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.718105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.718106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.718244 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:12.718300 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:12.718712 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 00:14:12.718802 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:12.719717 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.707405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38005 (local address 127.31.227.3:58901)
0812 00:14:12.707865 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:12.707913 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:12.708615 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:12.709011 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:12.709015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:12.709289 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:12.709620 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:12.709627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.710664 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.710665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:12.711974 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:12.711977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.712570 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.712571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:12.712723 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.713408 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:12.713420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:12.715215 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:12.717250 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:12.717259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:12.717262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:12.717621 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:12.717863 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:12.717864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:12.717864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:12.718013 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:12.718336 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:12.718344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:12.718519 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 00:14:12.719278 (+ 759us) client_negotiation.cc:240] Negotiation successful
0812 00:14:12.719457 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":119,"threads_started":1}
I20260812 00:14:12.720407 22497 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38005
I20260812 00:14:12.720510 22497 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:12.720731 22497 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:12.721174 21665 ts_manager.cc:194] Registered new tserver with Master: 5d5ded3d07494d84a1b0bc1e90b233b7 (127.31.227.3:41137)
I20260812 00:14:12.721674 21665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:58901
I20260812 00:14:12.725984 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:14:12.726694 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 21816
I20260812 00:14:12.858033 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 22029
I20260812 00:14:12.876423 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 22278
I20260812 00:14:12.892956 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 21603
I20260812 00:14:12.905695 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38005
--webserver_interface=127.31.227.62
--webserver_port=35125
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:38005
--ranger_config_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:12.992578 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:12.992825 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:12.992862 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:12.994117 22503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:12.994194 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:12.994220 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:12.994251 22503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:12.994280 22503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:12.995946 22503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:38005
--ranger_config_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:12.996307 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:12.996758 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:13.002331 22511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.002355 22514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.002604 22503 server_base.cc:1034] running on GCE node
W20260812 00:14:13.002796 22516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.003083 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:13.003405 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:13.004580 22503 hybrid_clock.cc:648] HybridClock initialized: now 1786493653004557 us; error 39 us; skew 500 ppm
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:13.007479 22503 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:13.007916 22503 webserver.cc:460] Webserver started at http://127.31.227.62:35125/ using document root <none> and password file <none>
I20260812 00:14:13.008163 22503 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:13.008239 22503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:13.085834 22503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:13.086650 22534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.086835 22503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:13.086933 22503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "632cab685d244b52890f17e30b2714fe"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zr1t"
server_key: "83de9ad7a917a9893ef13bae3b666dfb"
server_key_iv: "c4a9c60a91536c7fa648997e5bbc8d91"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:13.087328 22503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:13.096153 22503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:13.098057 22503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:13.098222 22503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:13.103302 22503 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:38005
I20260812 00:14:13.103328 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:38005 every 8 connection(s)
I20260812 00:14:13.103729 22503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:13.105028 22626 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:14:13.105775 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Bootstrap starting.
I20260812 00:14:13.112846 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 22503
I20260812 00:14:13.113188 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:35709
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:38005
--webserver_port=44103
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:13.114238 22626 log.cc:824] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:13.116616 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:14:13.116958 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Bootstrap complete.
I20260812 00:14:13.121229 22626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:13.121673 22626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632cab685d244b52890f17e30b2714fe, State: Initialized, Role: FOLLOWER
I20260812 00:14:13.121920 22626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:13.122030 22626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:13.122118 22626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:13.122185 22626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:14:13.123879 22626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:13.124022 22626 leader_election.cc:302] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 632cab685d244b52890f17e30b2714fe; no voters:
I20260812 00:14:13.124256 22626 leader_election.cc:288] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:14:13.124328 22630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:14:13.124548 22630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [term 2 LEADER]: Becoming Leader. State: Replica: 632cab685d244b52890f17e30b2714fe, State: Running, Role: LEADER
I20260812 00:14:13.124675 22626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:13.124697 22630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } }
I20260812 00:14:13.125125 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "632cab685d244b52890f17e30b2714fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } } }
I20260812 00:14:13.125195 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:13.125402 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632cab685d244b52890f17e30b2714fe. Latest consensus state: current_term: 2 leader_uuid: "632cab685d244b52890f17e30b2714fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cab685d244b52890f17e30b2714fe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 38005 } } }
I20260812 00:14:13.125499 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:13.125679 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:13.126219 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:13.126421 22636 catalog_manager.cc:1259] Loaded cluster ID: c1d642ce1f1a48df9c2cc78df19128f8
I20260812 00:14:13.126510 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:13.126799 22626 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 00:14:13.128221 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:13.128545 22636 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Loaded TSK: 0
I20260812 00:14:13.128846 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:14:13.218281 22629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:13.218509 22629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:13.218551 22629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:13.220057 22629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:13.220126 22629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:13.220172 22629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:14:13.222003 22629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:35709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=44103
--tserver_master_addrs=127.31.227.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:13.222301 22629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:13.222574 22629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:13.228351 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.228327 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.228331 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.229318 22629 server_base.cc:1034] running on GCE node
I20260812 00:14:13.229595 22629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:13.229910 22629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:13.237239 22629 hybrid_clock.cc:648] HybridClock initialized: now 1786493653237213 us; error 30 us; skew 500 ppm
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:13.240921 22629 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:14:13.241495 22629 webserver.cc:460] Webserver started at http://127.31.227.1:44103/ using document root <none> and password file <none>
I20260812 00:14:13.241755 22629 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:13.241863 22629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:13.268091 22629 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 00:14:13.269217 22700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.269428 22629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:13.269532 22629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "93158a7ec913495c9e506ff62c4a8565"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "c837c6620fe1e38b1f3c9df9cceaf37a"
server_key_iv: "9ac81205ff2bd164ff97e16960d3d370"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:13.270002 22629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:13.288928 22629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:13.291172 22629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:13.291406 22629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:13.291646 22629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:13.292050 22629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:13.292106 22629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.292177 22629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:13.292222 22629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.299633 22629 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:35709
I20260812 00:14:13.300143 22629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:14:13.300910 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 22629
I20260812 00:14:13.301213 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:43997
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:38005
--webserver_port=36285
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:13.303985 22888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:35709 every 8 connection(s)
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:13.321332 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.304671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38005 (local address 127.31.227.1:42741)
0812 00:14:13.306559 (+ 1888us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:13.306606 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:13.307137 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:13.307623 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:13.307627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:13.307862 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:13.308078 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:13.308083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.309310 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.309312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:13.310180 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:13.310182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.310843 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.310844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:13.310982 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.311528 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:13.311537 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:13.312805 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:13.315633 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:13.315640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:13.315642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:13.316109 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:13.316418 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:13.316419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:13.316419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:13.320103 (+ 3684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:13.320651 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:13.320659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:13.320820 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 00:14:13.320923 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 00:14:13.321093 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841,"thread_start_us":112,"threads_started":1}
I20260812 00:14:13.322186 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.304729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:42741 (local address 127.31.227.62:38005)
0812 00:14:13.304929 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:13.304940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:13.306731 (+ 1791us) server_negotiation.cc:407] Connection header received
0812 00:14:13.307368 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:13.307374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:13.307434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:13.307580 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:13.308210 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.309194 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.310280 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.310727 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.310853 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.312942 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:13.312947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:13.312947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:13.312959 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:13.315529 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.316210 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.316214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.316214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.316322 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.320202 (+ 3880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.320204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.320204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.320530 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:13.320617 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:13.321818 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0812 00:14:13.321950 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260812 00:14:13.323465 22889 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38005
I20260812 00:14:13.323567 22889 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.323757 22889 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
W20260812 00:14:13.324719 22889 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:38005 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:14:13.324800 22889 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.324862 22889 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:13.325263 22889 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.325352 22889 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
W20260812 00:14:13.325717 22889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:14:13.418152 22892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:13.418359 22892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:13.418383 22892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:13.420029 22892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:13.420094 22892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:13.420136 22892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:14:13.422336 22892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:43997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=36285
--tserver_master_addrs=127.31.227.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:13.422613 22892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:13.422925 22892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:13.429190 22904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.430212 22907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.430519 22905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.430900 22892 server_base.cc:1034] running on GCE node
I20260812 00:14:13.431135 22892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:13.431406 22892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:13.434677 22892 hybrid_clock.cc:648] HybridClock initialized: now 1786493653434637 us; error 51 us; skew 500 ppm
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:13.438169 22892 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:14:13.438742 22892 webserver.cc:460] Webserver started at http://127.31.227.2:36285/ using document root <none> and password file <none>
I20260812 00:14:13.439006 22892 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:13.439082 22892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:13.458406 22892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 00:14:13.459227 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.459396 22892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:13.459497 22892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "bcbdf5046a704a10875530956ed6c9cf"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3ea24edd4df9cfeda970dcf2ed23dc15"
server_key_iv: "fad1554824a39d8e45ff6697411cebcb"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:13.460743 22892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:13.474604 22892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:13.476933 22892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:13.477105 22892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:13.477350 22892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:13.477783 22892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:13.477838 22892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.477880 22892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:13.477908 22892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.484918 22892 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:43997
I20260812 00:14:13.485476 22892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:14:13.485973 23106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:43997 every 8 connection(s)
I20260812 00:14:13.491443 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 22892
I20260812 00:14:13.491775 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.227.20:52521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:41137
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:38005
--webserver_port=36423
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:13.497920 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.486843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:58813 (local address 127.31.227.62:38005)
0812 00:14:13.486908 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:13.486910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:13.487221 (+ 311us) server_negotiation.cc:407] Connection header received
0812 00:14:13.487915 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:13.487917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:13.487933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:13.488017 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:13.488731 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.489504 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.491089 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.491305 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.491391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.493688 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:13.493690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:13.493691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:13.493699 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:13.495524 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.496714 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.496716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.496716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.496781 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.497087 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.497089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.497089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.497236 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:13.497283 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:13.497722 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 00:14:13.497819 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:13.498585 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.486830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38005 (local address 127.31.227.2:58813)
0812 00:14:13.487083 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:13.487123 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:13.487772 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:13.488098 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:13.488102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:13.488332 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:13.488615 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:13.488622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.489661 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.489662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:13.490987 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:13.490990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.491423 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.491425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:13.491607 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.492172 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:13.492181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:13.493567 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:13.496305 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:13.496312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:13.496349 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:13.496598 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:13.496858 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:13.496859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:13.496860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:13.496995 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:13.497336 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:13.497342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:13.497539 (+ 197us) client_negotiation.cc:769] Sending connection context
0812 00:14:13.498148 (+ 609us) client_negotiation.cc:240] Negotiation successful
0812 00:14:13.498326 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
I20260812 00:14:13.499315 23107 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38005
I20260812 00:14:13.499419 23107 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.499627 23107 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
W20260812 00:14:13.500313 23107 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:38005 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:14:13.500381 23107 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.500442 23107 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:13.500831 23107 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.500905 23107 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
W20260812 00:14:13.501232 23107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:14:13.619834 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:13.620158 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:13.620199 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:13.621907 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:13.621978 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:13.622020 23112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:14:13.624300 23112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.227.20:52521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:41137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=36423
--tserver_master_addrs=127.31.227.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:13.624756 23112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:13.625063 23112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:13.631856 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.631884 23121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.632267 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.632954 23112 server_base.cc:1034] running on GCE node
I20260812 00:14:13.633190 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:13.633504 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:13.634681 23112 hybrid_clock.cc:648] HybridClock initialized: now 1786493653634650 us; error 42 us; skew 500 ppm
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:13.638188 23112 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:14:13.638749 23112 webserver.cc:460] Webserver started at http://127.31.227.3:36423/ using document root <none> and password file <none>
I20260812 00:14:13.639050 23112 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:13.639137 23112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:13.657752 23112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:13.658738 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.658918 23112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:13.659021 23112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "5d5ded3d07494d84a1b0bc1e90b233b7"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zr1t"
server_key: "3a812612f839ac0c34aca4e327eea13e"
server_key_iv: "a7ae1afadd07d14ae9a6d869a720b372"
server_key_version: "kuduclusterkey@0"
I20260812 00:14:13.659420 23112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:13.674592 23112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:13.677191 23112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:13.677416 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:13.677747 23112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:13.678153 23112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:13.678221 23112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.678292 23112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:13.678340 23112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.686395 23112 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41137
I20260812 00:14:13.686514 23311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41137 every 8 connection(s)
I20260812 00:14:13.686864 23112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:14:13.692319 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 23112
Aug 12 00:14:13 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:13.697798 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.687698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:40883 (local address 127.31.227.62:38005)
0812 00:14:13.687753 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:13.687756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:13.687998 (+ 242us) server_negotiation.cc:407] Connection header received
0812 00:14:13.688522 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:13.688523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:13.688538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:13.688603 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:13.689176 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.689821 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.691030 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.691282 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.691358 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.694371 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:13.694373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:13.694373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:13.694381 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:13.696125 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.696614 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.696616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.696616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.696675 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.696925 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.696926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.696927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.697069 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:13.697138 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:13.697582 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 00:14:13.697691 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:13.698592 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.687712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:38005 (local address 127.31.227.3:40883)
0812 00:14:13.687879 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:13.687922 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:13.688408 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:13.688678 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:13.688682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:13.688870 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:13.689071 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:13.689075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.689926 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.689927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:13.690922 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:13.690925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.691406 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.691408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:13.691540 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.692220 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:13.692231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:13.694265 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:13.696203 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:13.696211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:13.696214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:13.696517 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:13.696747 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:13.696748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:13.696748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:13.696845 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:13.697169 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:13.697176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:13.697349 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 00:14:13.698188 (+ 839us) client_negotiation.cc:240] Negotiation successful
0812 00:14:13.698356 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260812 00:14:13.699194 23312 heartbeater.cc:347] Connected to a master server at 127.31.227.62:38005
I20260812 00:14:13.699290 23312 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.699832 23312 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
W20260812 00:14:13.700954 23312 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:38005 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:14:13.701026 23312 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.701092 23312 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:13.701485 23312 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:13.701578 23312 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
W20260812 00:14:13.701923 23312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:14:13.703075 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.692766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.31.227.62:38005)
0812 00:14:13.692904 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:13.692906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:13.693193 (+ 287us) server_negotiation.cc:407] Connection header received
0812 00:14:13.693214 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:13.693215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:13.693226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:13.693316 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:13.694011 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.694744 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.696221 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.696528 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.696588 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.699078 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:13.699079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:13.699079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:13.699087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:13.700920 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.702188 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.702190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.702190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.702249 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:13.702491 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:13.702492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:13.702492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:13.702619 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:13.702809 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:13.702858 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 00:14:13.702944 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
Aug 12 00:14:14 dist-test-slave-zr1t krb5kdc[20830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:14.325886 22889 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:14.326053 22889 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:14.326493 22562 ts_manager.cc:194] Registered new tserver with Master: 93158a7ec913495c9e506ff62c4a8565 (127.31.227.1:35709)
I20260812 00:14:14.327214 22562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:42741
I20260812 00:14:14.501467 23107 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:14.501659 23107 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:14.502027 22562 ts_manager.cc:194] Registered new tserver with Master: bcbdf5046a704a10875530956ed6c9cf (127.31.227.2:43997)
I20260812 00:14:14.502579 22562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:58813
Aug 12 00:14:14 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:14.702127 23312 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:14.702303 23312 heartbeater.cc:511] Master 127.31.227.62:38005 requested a full tablet report, sending...
I20260812 00:14:14.702661 22562 ts_manager.cc:194] Registered new tserver with Master: 5d5ded3d07494d84a1b0bc1e90b233b7 (127.31.227.3:41137)
I20260812 00:14:14.703205 22562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:40883
I20260812 00:14:14.709233 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:14:14.721716 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.710332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.31.227.62:38005)
0812 00:14:14.710484 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.710487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.713707 (+ 3220us) server_negotiation.cc:407] Connection header received
0812 00:14:14.713779 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.713786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.713801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.713884 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:14.714619 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.715409 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.716471 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.716779 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.716853 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.718482 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:14.718484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:14.718485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:14.718493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:14.720343 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:14.720766 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:14.720768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:14.720768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:14.720844 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:14.721090 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:14.721091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:14.721091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:14.721230 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:14.721331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.721482 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.721609 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260812 00:14:14.724045 22562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:14:14.726372 22562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:14:14.738194 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.730187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41137 (local address 127.0.0.1:60390)
0812 00:14:14.733657 (+ 3470us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.733669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.733734 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.734044 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.734046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.734048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.734096 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.734097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.735371 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.735372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.736661 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.736665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.737845 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.737846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.737939 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.737964 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.738044 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.738102 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3451,"thread_start_us":72,"threads_started":1}
I20260812 00:14:14.738656 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.730369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.31.227.3:41137)
0812 00:14:14.730492 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.730497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.733801 (+ 3304us) server_negotiation.cc:407] Connection header received
0812 00:14:14.733843 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.733845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.733862 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.733936 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.734179 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.735270 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.736778 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.737658 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.737733 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.737802 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.738532 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.738577 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260812 00:14:14.738735 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.729695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35709 (local address 127.0.0.1:47208)
0812 00:14:14.730615 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.730625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.730671 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.733945 (+ 3274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.733949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.733952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.734028 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.734029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.735641 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.735642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.736791 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.736793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.738491 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.738492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.738562 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.738566 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.738617 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.738657 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":75,"threads_started":1}
I20260812 00:14:14.739696 23215 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6052c68eda4d13bbef30f4e4779e66 (DEFAULT_TABLE table=test-table [id=17472f25e1004d37abc2007c2e9ded4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:14.740120 23215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6052c68eda4d13bbef30f4e4779e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:14.741914 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.729927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.31.227.1:35709)
0812 00:14:14.733661 (+ 3734us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.733667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.733680 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:14:14.733720 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.733722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.733740 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.733832 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.734107 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.735545 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.736891 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.738320 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.738400 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.738450 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.738813 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.741826 (+ 3013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3694,"thread_start_us":74,"threads_started":1}
I20260812 00:14:14.744719 22777 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6052c68eda4d13bbef30f4e4779e66 (DEFAULT_TABLE table=test-table [id=17472f25e1004d37abc2007c2e9ded4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:14.745088 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6052c68eda4d13bbef30f4e4779e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:14.745872 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.729429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:43997 (local address 127.0.0.1:35456)
0812 00:14:14.729850 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.729893 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.729976 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.742455 (+ 12479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.742456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.742460 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.742508 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.742510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.743710 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.743711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.744718 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.744720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.745668 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.745669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.745729 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.745733 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.745762 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.745799 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":115,"threads_started":1}
I20260812 00:14:14.746272 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.729574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.31.227.2:43997)
0812 00:14:14.742238 (+ 12664us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.742244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.742256 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:14.742312 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.742315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.742332 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.742586 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.742672 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.743605 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.744823 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.745556 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.746061 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.746130 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.746158 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.746197 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12622,"thread_start_us":70,"threads_started":1}
I20260812 00:14:14.748486 23370 tablet_bootstrap.cc:492] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: Bootstrap starting.
I20260812 00:14:14.749130 23370 tablet_bootstrap.cc:654] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:14.749466 23370 log.cc:824] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:14.749550 23368 tablet_bootstrap.cc:492] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Bootstrap starting.
I20260812 00:14:14.749527 22992 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6052c68eda4d13bbef30f4e4779e66 (DEFAULT_TABLE table=test-table [id=17472f25e1004d37abc2007c2e9ded4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:14.749907 22992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6052c68eda4d13bbef30f4e4779e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:14.750313 23368 tablet_bootstrap.cc:654] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:14.750339 23370 tablet_bootstrap.cc:492] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: No bootstrap required, opened a new log
I20260812 00:14:14.750401 23370 ts_tablet_manager.cc:1397] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 00:14:14.750612 23368 log.cc:824] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:14.751389 23368 tablet_bootstrap.cc:492] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: No bootstrap required, opened a new log
I20260812 00:14:14.751456 23368 ts_tablet_manager.cc:1397] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:14:14.752799 23370 raft_consensus.cc:348] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.752969 23370 raft_consensus.cc:374] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:14.753002 23370 raft_consensus.cc:729] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93158a7ec913495c9e506ff62c4a8565, State: Initialized, Role: FOLLOWER
I20260812 00:14:14.753113 23370 consensus_queue.cc:260] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.753351 23370 ts_tablet_manager.cc:1428] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:14:14.753475 22889 heartbeater.cc:503] Master 127.31.227.62:38005 was elected leader, sending a full tablet report...
I20260812 00:14:14.754344 23380 tablet_bootstrap.cc:492] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Bootstrap starting.
I20260812 00:14:14.754806 23368 raft_consensus.cc:348] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.755025 23380 tablet_bootstrap.cc:654] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:14.755070 23368 raft_consensus.cc:374] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:14.755132 23368 raft_consensus.cc:729] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5ded3d07494d84a1b0bc1e90b233b7, State: Initialized, Role: FOLLOWER
I20260812 00:14:14.755288 23368 consensus_queue.cc:260] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.755577 23368 ts_tablet_manager.cc:1428] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260812 00:14:14.755560 23380 log.cc:824] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:14.755698 23312 heartbeater.cc:503] Master 127.31.227.62:38005 was elected leader, sending a full tablet report...
I20260812 00:14:14.756749 23380 tablet_bootstrap.cc:492] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: No bootstrap required, opened a new log
I20260812 00:14:14.756824 23380 ts_tablet_manager.cc:1397] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:14:14.758448 23380 raft_consensus.cc:348] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.758569 23380 raft_consensus.cc:374] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:14.758589 23380 raft_consensus.cc:729] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbdf5046a704a10875530956ed6c9cf, State: Initialized, Role: FOLLOWER
I20260812 00:14:14.758704 23380 consensus_queue.cc:260] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.758962 23380 ts_tablet_manager.cc:1428] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:14:14.758994 23107 heartbeater.cc:503] Master 127.31.227.62:38005 was elected leader, sending a full tablet report...
W20260812 00:14:14.806056 22891 tablet.cc:2369] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:14:14.866106 23381 raft_consensus.cc:482] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:14.866279 23381 raft_consensus.cc:504] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.866707 23381 leader_election.cc:288] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcbdf5046a704a10875530956ed6c9cf (127.31.227.2:43997), 93158a7ec913495c9e506ff62c4a8565 (127.31.227.1:35709)
I20260812 00:14:14.867569 23384 raft_consensus.cc:482] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:14.867681 23384 raft_consensus.cc:504] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.868018 23384 leader_election.cc:288] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d5ded3d07494d84a1b0bc1e90b233b7 (127.31.227.3:41137), 93158a7ec913495c9e506ff62c4a8565 (127.31.227.1:35709)
I20260812 00:14:14.871091 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.867046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:54047 (local address 127.31.227.2:43997)
0812 00:14:14.867098 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.867100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.867294 (+ 194us) server_negotiation.cc:407] Connection header received
0812 00:14:14.867428 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.867430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.867442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.867480 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.867720 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.868468 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.870110 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.870675 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.870726 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.870735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.870983 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.871028 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:14:14.871482 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.866933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:43997 (local address 127.31.227.3:54047)
0812 00:14:14.867211 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.867223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.867333 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.867557 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.867557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.867559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.867643 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.867645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.868689 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.868690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.870001 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.870005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.870820 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.870821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.870901 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.870905 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.871344 (+ 439us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.871389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":125,"threads_started":1}
I20260812 00:14:14.871903 23017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6052c68eda4d13bbef30f4e4779e66" candidate_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbdf5046a704a10875530956ed6c9cf" is_pre_election: true
I20260812 00:14:14.872012 23017 raft_consensus.cc:2454] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5ded3d07494d84a1b0bc1e90b233b7 in term 0.
I20260812 00:14:14.872325 23147 leader_election.cc:302] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5ded3d07494d84a1b0bc1e90b233b7, bcbdf5046a704a10875530956ed6c9cf; no voters:
I20260812 00:14:14.872457 23381 raft_consensus.cc:2781] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:14:14.872493 23381 raft_consensus.cc:482] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:14:14.872512 23381 raft_consensus.cc:3037] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:14.873422 23381 raft_consensus.cc:504] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.873564 23381 leader_election.cc:288] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [CANDIDATE]: Term 1 election: Requested vote from peers bcbdf5046a704a10875530956ed6c9cf (127.31.227.2:43997), 93158a7ec913495c9e506ff62c4a8565 (127.31.227.1:35709)
I20260812 00:14:14.873873 23017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6052c68eda4d13bbef30f4e4779e66" candidate_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbdf5046a704a10875530956ed6c9cf"
I20260812 00:14:14.873946 23017 raft_consensus.cc:3037] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:14.874742 23017 raft_consensus.cc:2454] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5ded3d07494d84a1b0bc1e90b233b7 in term 1.
I20260812 00:14:14.874979 23147 leader_election.cc:302] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5ded3d07494d84a1b0bc1e90b233b7, bcbdf5046a704a10875530956ed6c9cf; no voters:
I20260812 00:14:14.875095 23381 raft_consensus.cc:2781] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:14.875218 23381 raft_consensus.cc:686] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 1 LEADER]: Becoming Leader. State: Replica: 5d5ded3d07494d84a1b0bc1e90b233b7, State: Running, Role: LEADER
I20260812 00:14:14.875301 23381 consensus_queue.cc:237] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } }
I20260812 00:14:14.876190 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.868414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:43313 (local address 127.31.227.1:35709)
0812 00:14:14.869136 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.869138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.869145 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:14:14.869171 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.869172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.869182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.869242 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.869789 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.870513 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.874230 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.875098 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.875188 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.875198 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.876049 (+ 851us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.876094 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":86,"threads_started":1}
I20260812 00:14:14.876377 22562 catalog_manager.cc:5495] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d5ded3d07494d84a1b0bc1e90b233b7 (127.31.227.3). New cstate: current_term: 1 leader_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 } health_report { overall_health: UNKNOWN } } }
I20260812 00:14:14.877696 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.868229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35709 (local address 127.31.227.2:43313)
0812 00:14:14.868875 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.868885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.868937 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.869291 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.869292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.869294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.869375 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.869377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.870619 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.870620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.871843 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.871847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.875250 (+ 3403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.875251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.875323 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.875327 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.875376 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.875414 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":94,"threads_started":1}
I20260812 00:14:14.878176 22791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6052c68eda4d13bbef30f4e4779e66" candidate_uuid: "bcbdf5046a704a10875530956ed6c9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93158a7ec913495c9e506ff62c4a8565" is_pre_election: true
I20260812 00:14:14.878327 22791 raft_consensus.cc:2454] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbdf5046a704a10875530956ed6c9cf in term 0.
I20260812 00:14:14.878593 22924 leader_election.cc:302] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93158a7ec913495c9e506ff62c4a8565, bcbdf5046a704a10875530956ed6c9cf; no voters:
I20260812 00:14:14.878720 23384 raft_consensus.cc:2741] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260812 00:14:14.879608 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.867400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:34211 (local address 127.31.227.1:35709)
0812 00:14:14.867449 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.867451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.874049 (+ 6598us) server_negotiation.cc:407] Connection header received
0812 00:14:14.874152 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.874153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.874163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.874193 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.875507 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.876161 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.878757 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.879251 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.879298 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.879305 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.879511 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.879549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:14:14.879791 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.867196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35709 (local address 127.31.227.3:34211)
0812 00:14:14.873988 (+ 6792us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.873999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.874078 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.874435 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.874435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.874438 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.874504 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.874506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.877087 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.877088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.878649 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.878653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.879366 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.879367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.879435 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.879439 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.879678 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.879719 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6774,"thread_start_us":79,"threads_started":1}
I20260812 00:14:14.880105 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6052c68eda4d13bbef30f4e4779e66" candidate_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93158a7ec913495c9e506ff62c4a8565"
I20260812 00:14:14.880192 22790 raft_consensus.cc:3037] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:14.880313 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.868771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:36463 (local address 127.31.227.3:41137)
0812 00:14:14.868806 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.868808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.869057 (+ 249us) server_negotiation.cc:407] Connection header received
0812 00:14:14.869545 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.869546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.869556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.869585 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:14.871977 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.872605 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.876776 (+ 4171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.877318 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.877390 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.877398 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.880225 (+ 2827us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.880263 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:14:14.880519 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.868621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41137 (local address 127.31.227.2:36463)
0812 00:14:14.869010 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.869018 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:14.869448 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:14.869659 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:14.869660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:14.869662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:14.869719 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.869721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.872709 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.872710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.873850 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:14.873863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.880055 (+ 6192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.880057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:14.880138 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.880143 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:14:14.880404 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 00:14:14.880447 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":74,"threads_started":1}
I20260812 00:14:14.880828 23243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6052c68eda4d13bbef30f4e4779e66" candidate_uuid: "bcbdf5046a704a10875530956ed6c9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" is_pre_election: true
I20260812 00:14:14.880954 22791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6052c68eda4d13bbef30f4e4779e66" candidate_uuid: "5d5ded3d07494d84a1b0bc1e90b233b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93158a7ec913495c9e506ff62c4a8565" is_pre_election: true
I20260812 00:14:14.881155 22790 raft_consensus.cc:2454] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5ded3d07494d84a1b0bc1e90b233b7 in term 1.
I20260812 00:14:14.885428 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.882278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.31.227.3:41137)
0812 00:14:14.882332 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:14.882334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:14.882343 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:14.882421 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:14.882422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:14.882440 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:14.882510 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:14.882677 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.883341 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.884426 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:14.884681 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:14.884748 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:14.884966 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:14.885091 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:14.885169 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:14.885345 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 00:14:14.885363 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:14:14.887136 22790 raft_consensus.cc:1261] T 6b6052c68eda4d13bbef30f4e4779e66 P 93158a7ec913495c9e506ff62c4a8565 [term 1 FOLLOWER]: Refusing update from remote peer 5d5ded3d07494d84a1b0bc1e90b233b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:14.887403 23381 consensus_queue.cc:1035] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93158a7ec913495c9e506ff62c4a8565" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:14.888068 23017 raft_consensus.cc:1261] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 1 FOLLOWER]: Refusing update from remote peer 5d5ded3d07494d84a1b0bc1e90b233b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:14.888652 23381 consensus_queue.cc:1035] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbdf5046a704a10875530956ed6c9cf" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:14.893072 23404 mvcc.cc:205] Tried to move back new op lower bound from 7317478010415448064 to 7317478010369728512. Current Snapshot: MvccSnapshot[applied={T|T < 7317478010415448064}]
I20260812 00:14:14.895814 22562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:14.895917 22562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:14.896232 23405 mvcc.cc:205] Tried to move back new op lower bound from 7317478010415448064 to 7317478010369728512. Current Snapshot: MvccSnapshot[applied={T|T < 7317478010415448064}]
I20260812 00:14:14.896849 22562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 632cab685d244b52890f17e30b2714fe: Sending DeleteTablet for 3 replicas of tablet 6b6052c68eda4d13bbef30f4e4779e66
I20260812 00:14:14.896919 23407 mvcc.cc:205] Tried to move back new op lower bound from 7317478010415448064 to 7317478010369728512. Current Snapshot: MvccSnapshot[applied={T|T < 7317478010415448064}]
I20260812 00:14:14.897293 23215 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6052c68eda4d13bbef30f4e4779e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:14 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260812 00:14:14.897938 22992 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6052c68eda4d13bbef30f4e4779e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:14 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:35456
I20260812 00:14:14.898140 22777 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6052c68eda4d13bbef30f4e4779e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:14 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:47208
I20260812 00:14:14.898617 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 22629
I20260812 00:14:14.915124 23416 tablet_replica.cc:330] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: stopping tablet replica
I20260812 00:14:14.920940 23416 raft_consensus.cc:2229] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:14:14.921646 23415 tablet_replica.cc:330] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: stopping tablet replica
I20260812 00:14:14.921499 23416 raft_consensus.cc:2258] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:14.922031 23415 raft_consensus.cc:2229] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:14:14.922268 23415 raft_consensus.cc:2258] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:14.923085 23415 ts_tablet_manager.cc:1905] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:14.926664 23416 ts_tablet_manager.cc:1905] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:14.928476 23416 ts_tablet_manager.cc:1918] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:14.928644 23416 log.cc:1196] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/6b6052c68eda4d13bbef30f4e4779e66
I20260812 00:14:14.929028 23416 ts_tablet_manager.cc:1939] T 6b6052c68eda4d13bbef30f4e4779e66 P bcbdf5046a704a10875530956ed6c9cf: Deleting consensus metadata
I20260812 00:14:14.929160 23415 ts_tablet_manager.cc:1918] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:14.929785 23415 log.cc:1196] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/6b6052c68eda4d13bbef30f4e4779e66
I20260812 00:14:14.930189 23415 ts_tablet_manager.cc:1939] T 6b6052c68eda4d13bbef30f4e4779e66 P 5d5ded3d07494d84a1b0bc1e90b233b7: Deleting consensus metadata
I20260812 00:14:14.930529 22538 catalog_manager.cc:4841] TS bcbdf5046a704a10875530956ed6c9cf (127.31.227.2:43997): tablet 6b6052c68eda4d13bbef30f4e4779e66 (table test-table [id=17472f25e1004d37abc2007c2e9ded4e]) successfully deleted
I20260812 00:14:14.931133 22545 catalog_manager.cc:4841] TS 5d5ded3d07494d84a1b0bc1e90b233b7 (127.31.227.3:41137): tablet 6b6052c68eda4d13bbef30f4e4779e66 (table test-table [id=17472f25e1004d37abc2007c2e9ded4e]) successfully deleted
W20260812 00:14:14.931638 22538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:35709 (error 108)
I20260812 00:14:14.932646 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.932483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35709 (local address 127.0.0.1:47216)
0812 00:14:14.932546 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.932607 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 00:14:14.932647 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 22892
W20260812 00:14:14.940321 22538 catalog_manager.cc:4584] TS 93158a7ec913495c9e506ff62c4a8565 (127.31.227.1:35709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b6052c68eda4d13bbef30f4e4779e66: Network error: Client connection negotiation failed: client connection to 127.31.227.1:35709: connect: Connection refused (error 111)
I20260812 00:14:14.990681 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 23112
I20260812 00:14:14.993034 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:14.992873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35709 (local address 127.0.0.1:47220)
0812 00:14:14.992952 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:14.992985 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Aug 12 00:14:15 dist-test-slave-zr1t krb5kdc[20830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for HTTP/127.31.227.20@KRBTEST.COM
I20260812 00:14:15.024675 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:15.024521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:35709 (local address 127.0.0.1:47236)
0812 00:14:15.024593 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:15.024624 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 00:14:15.034099 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 22503
I20260812 00:14:15.048779 32652 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 00:14:15.398033 32652 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 00:14:15.398125 32652 mini_ranger.cc:66] Stopping Ranger...
I20260812 00:14:15.753273 32652 mini_ranger.cc:68] Stopped Ranger
2026-08-12 00:14:15.753 UTC [20882] LOG: received smart shutdown request
2026-08-12 00:14:15.758 UTC [20882] LOG: background worker "logical replication launcher" (PID 20900) exited with exit code 1
2026-08-12 00:14:15.758 UTC [20886] LOG: shutting down
2026-08-12 00:14:15.899 UTC [20882] LOG: database system is shut down
2026-08-12T00:14:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49441 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:16 dist-test-slave-zr1t krb5kdc[23440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:16 dist-test-slave-zr1t krb5kdc[23440](info): set up 2 sockets
Aug 12 00:14:16 dist-test-slave-zr1t krb5kdc[23440](info): commencing operation
krb5kdc: starting...
W20260812 00:14:17.295105 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:14:17.417373 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:44625
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:44625,127.31.227.61:32829,127.31.227.60:39739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:17.505822 23463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:17.506021 23463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:17.506043 23463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:17.507166 23463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:17.507207 23463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:17.507221 23463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:17.507269 23463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:17.507287 23463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:17.508749 23463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:44625,127.31.227.61:32829,127.31.227.60:39739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:17.509014 23463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:17.509250 23463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:17.514223 23478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:17.514238 23471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:17.514259 23473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:17.514477 23463 server_base.cc:1034] running on GCE node
I20260812 00:14:17.514724 23463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:17.514971 23463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:17.516110 23463 hybrid_clock.cc:648] HybridClock initialized: now 1786493657516096 us; error 30 us; skew 500 ppm
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:17.521596 23463 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:17.522060 23463 webserver.cc:460] Webserver started at http://127.31.227.62:35973/ using document root <none> and password file <none>
I20260812 00:14:17.522243 23463 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:17.522298 23463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:17.522388 23463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:17.523391 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "748e4f00947d47bfb731938bba209b94"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "a729d89abeb65c31e6a612253bf2785d"
server_key_iv: "b5f892173147f301904410f74e805c8a"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.523689 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "748e4f00947d47bfb731938bba209b94"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "a729d89abeb65c31e6a612253bf2785d"
server_key_iv: "b5f892173147f301904410f74e805c8a"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.524959 23463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:17.525681 23488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:17.525852 23463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:17.525938 23463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "748e4f00947d47bfb731938bba209b94"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "a729d89abeb65c31e6a612253bf2785d"
server_key_iv: "b5f892173147f301904410f74e805c8a"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.525993 23463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:17.545347 23463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:17.547173 23463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:17.547281 23463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:17.551200 23463 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:44625
I20260812 00:14:17.551293 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:44625 every 8 connection(s)
I20260812 00:14:17.551698 23463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:17.552595 23571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:17.553637 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 23463
I20260812 00:14:17.553766 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:14:17.553978 32652 external_mini_cluster.cc:1428] Setting key 8d03f2b0949c761bcc8c380f11d85277
I20260812 00:14:17.554680 23571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.227.61@KRBTEST.COM; defaulting to no policy
I20260812 00:14:17.564502 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.555511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.31.227.62:44625)
0812 00:14:17.556035 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.556041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.556076 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:14:17.556458 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.556460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.556480 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.556584 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.557104 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.557965 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.558886 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.559199 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.559324 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.561317 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.561329 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.561329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.561335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.562904 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.563380 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.563383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.563383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.563466 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.563707 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.563708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.563708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.563908 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.564043 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.564214 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.564416 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":492,"thread_start_us":62,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.61@KRBTEST.COM" created.
I20260812 00:14:17.564512 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.555444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:44625 (local address 127.0.0.1:59016)
0812 00:14:17.555658 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.555701 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.556287 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.556588 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.556590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.556850 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.557000 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.557005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.558073 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.558074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.558785 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.558788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.559299 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.559300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.559367 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.559911 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.559926 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.561214 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.563074 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.563081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.563083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.563280 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.563546 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.563547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.563547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.563634 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.563996 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.564001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.564064 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.564157 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.564298 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260812 00:14:17.565438 23571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:17.565788 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.565691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43862)
0812 00:14:17.565730 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.565770 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 00:14:17.565908 23494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
I20260812 00:14:17.566313 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.566214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43866)
0812 00:14:17.566269 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.566292 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 00:14:17.566471 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:32829: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab.
Entry for principal kudu/127.31.227.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab.
I20260812 00:14:17.578003 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:32829
--webserver_interface=127.31.227.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab
--principal=kudu/127.31.227.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:44625,127.31.227.61:32829,127.31.227.60:39739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:17.615728 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } attempt: 1
I20260812 00:14:17.616297 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.616148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43880)
0812 00:14:17.616220 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.616261 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 00:14:17.616691 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.616579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43894)
0812 00:14:17.616629 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.616669 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:14:17.616802 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:32829: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
W20260812 00:14:17.663040 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:17.663280 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:17.663328 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:17.664481 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:17.664557 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:17.664585 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:17.664616 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:17.664639 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:17.666208 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:44625,127.31.227.61:32829,127.31.227.60:39739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab
--principal=kudu/127.31.227.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.227.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:17.666499 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:17.666731 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:17.671784 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:17.671763 23591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:17.671788 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:17.672050 23584 server_base.cc:1034] running on GCE node
I20260812 00:14:17.672305 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:17.672657 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:17.673843 23584 hybrid_clock.cc:648] HybridClock initialized: now 1786493657673831 us; error 27 us; skew 500 ppm
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:17.676304 23584 init.cc:377] Logged in from keytab as kudu/127.31.227.61@KRBTEST.COM (short username kudu)
I20260812 00:14:17.676791 23584 webserver.cc:460] Webserver started at http://127.31.227.61:39943/ using document root <none> and password file <none>
I20260812 00:14:17.677040 23584 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:17.677111 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:17.677266 23584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:17.678418 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data/instance:
uuid: "e8db24742fd744fca990025a8b08c87b"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "0dc5ea9b0cd5b01a5824cd747d5a6597"
server_key_iv: "093696b00ac94779f029c2a65821e166"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.678777 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal/instance:
uuid: "e8db24742fd744fca990025a8b08c87b"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "0dc5ea9b0cd5b01a5824cd747d5a6597"
server_key_iv: "093696b00ac94779f029c2a65821e166"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.680114 23584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:14:17.680922 23608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:17.681114 23584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:17.681219 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal
uuid: "e8db24742fd744fca990025a8b08c87b"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "0dc5ea9b0cd5b01a5824cd747d5a6597"
server_key_iv: "093696b00ac94779f029c2a65821e166"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.681304 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:17.684962 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } attempt: 2
I20260812 00:14:17.685393 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.685251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43908)
0812 00:14:17.685314 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.685356 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:14:17.685783 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.685687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43912)
0812 00:14:17.685731 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.685760 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 00:14:17.685926 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:32829: Network error: Client connection negotiation failed: client connection to 127.31.227.61:32829: connect: Connection refused (error 111)
I20260812 00:14:17.691766 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:17.693704 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:17.693864 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:17.698206 23584 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.61:32829
I20260812 00:14:17.698261 23691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.61:32829 every 8 connection(s)
I20260812 00:14:17.698617 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
I20260812 00:14:17.699462 23697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:17.701016 23697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:17.702984 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 23584
I20260812 00:14:17.703078 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-1/wal/instance
I20260812 00:14:17.703285 32652 external_mini_cluster.cc:1428] Setting key 27efc0b126ff9a30720ee75e57704fbd
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:17.710317 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.701780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.31.227.62:44625)
0812 00:14:17.701870 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.701872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.702025 (+ 153us) server_negotiation.cc:407] Connection header received
0812 00:14:17.702510 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.702512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.702524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.702577 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.703272 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.703832 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.704849 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.705086 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.705177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.707328 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.707330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.707330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.707338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.708801 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.709242 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.709243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.709244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.709311 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.709637 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.709639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.709639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.709766 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.709849 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.710135 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.710229 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 00:14:17.710525 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.701738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:44625 (local address 127.0.0.1:59028)
0812 00:14:17.701900 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.701938 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.702426 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.702629 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.702632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.702877 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.703163 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.703176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.703941 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.703943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.704752 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.704754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.705189 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.705190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.705320 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.705891 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.705899 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.707226 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.708912 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.708920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.708922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.709153 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.709387 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.709388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.709388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.709561 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.709839 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.709847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.709993 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.710101 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.710261 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260812 00:14:17.711090 23697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.227.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.60@KRBTEST.COM" created.
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:14:17.718911 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.711380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43928)
0812 00:14:17.711435 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.711447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.711488 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.711767 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.711768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.711916 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.711965 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.711966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.712904 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.712905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.713431 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.713433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.713961 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.713962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.714066 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.714419 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.714424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.715638 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.717837 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.717838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.717839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.718007 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.718231 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.718232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.718233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.718287 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.718585 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.718586 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.718624 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.718690 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.718788 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 00:14:17.718976 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.711468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.31.227.61:32829)
0812 00:14:17.711635 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.711640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.711656 (+ 16us) server_negotiation.cc:407] Connection header received
0812 00:14:17.711673 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.711675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.711685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.711766 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.712050 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.712790 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.713513 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.713845 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.713938 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.715739 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.715747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.715748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.715755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.717738 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.718099 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.718102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.718103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.718164 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.718359 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.718359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.718360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.718512 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.718606 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.718717 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.718842 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260812 00:14:17.719595 23697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:17.719966 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.719819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55044)
0812 00:14:17.719873 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.719931 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 00:14:17.720129 23610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
I20260812 00:14:17.720557 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.720433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55058)
0812 00:14:17.720495 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.720526 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 00:14:17.720678 23697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:39739: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab.
Entry for principal kudu/127.31.227.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab.
I20260812 00:14:17.730151 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:39739
--webserver_interface=127.31.227.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab
--principal=kudu/127.31.227.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:44625,127.31.227.61:32829,127.31.227.60:39739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:17.769904 23697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } attempt: 1
I20260812 00:14:17.770435 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.770285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55070)
0812 00:14:17.770347 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.770399 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:14:17.770848 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.770717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55078)
0812 00:14:17.770789 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.770821 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 00:14:17.771005 23697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:39739: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
I20260812 00:14:17.777102 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } attempt: 3
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:14:17.784335 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.777413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43930)
0812 00:14:17.777474 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.777486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.777568 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.777726 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.777727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.777871 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.777960 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.777962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.778916 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.778918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.779410 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.779412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.779863 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.779864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.779933 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.780386 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.780390 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.781559 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.783318 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.783319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.783319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.783494 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.783728 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.783729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.783729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.783786 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.784088 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.784089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.784138 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.784193 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.784271 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:14:17.784436 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.777486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.31.227.61:32829)
0812 00:14:17.777558 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.777560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.777574 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:14:17.777646 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.777647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.777659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.777711 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.778047 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.778805 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.779492 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.779786 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.779854 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.781665 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.781667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.781667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.781675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.783213 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.783570 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.783572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.783572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.783668 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.783851 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.783852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.783853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.783989 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.784081 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.784241 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.784322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:14:17.784869 23571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:17.785208 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.785091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55082)
0812 00:14:17.785136 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.785183 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 00:14:17.785612 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.785494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55098)
0812 00:14:17.785559 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.785585 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 00:14:17.785698 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:39739: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
W20260812 00:14:17.815608 23707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:17.815804 23707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:17.815828 23707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 00:14:17.816824 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } attempt: 1
W20260812 00:14:17.817076 23707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:17.817137 23707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:17.817162 23707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:17.817191 23707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:17.817214 23707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:17.817327 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.817218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55106)
0812 00:14:17.817262 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.817300 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260812 00:14:17.817771 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.817670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55114)
0812 00:14:17.817717 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.817742 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:14:17.817859 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:39739: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
I20260812 00:14:17.819022 23707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:44625,127.31.227.61:32829,127.31.227.60:39739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab
--principal=kudu/127.31.227.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.227.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:17.819286 23707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:17.819522 23707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:17.824467 23716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:17.824604 23707 server_base.cc:1034] running on GCE node
W20260812 00:14:17.824443 23719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:17.824450 23715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:17.824991 23707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:17.825289 23707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:17.826431 23707 hybrid_clock.cc:648] HybridClock initialized: now 1786493657826389 us; error 58 us; skew 500 ppm
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:17.829008 23707 init.cc:377] Logged in from keytab as kudu/127.31.227.60@KRBTEST.COM (short username kudu)
I20260812 00:14:17.829582 23707 webserver.cc:460] Webserver started at http://127.31.227.60:39041/ using document root <none> and password file <none>
I20260812 00:14:17.829805 23707 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:17.829850 23707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:17.830001 23707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:17.831056 23707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data/instance:
uuid: "aa1fd1ccf8dd420f826687f6a7242b03"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "ad1c261d4937501336c9a046cd6521df"
server_key_iv: "7053f05af4021070f0ae5501d3be206d"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.831439 23707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal/instance:
uuid: "aa1fd1ccf8dd420f826687f6a7242b03"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "ad1c261d4937501336c9a046cd6521df"
server_key_iv: "7053f05af4021070f0ae5501d3be206d"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.832752 23707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:17.833463 23729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:17.833652 23707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:14:17.833760 23707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal
uuid: "aa1fd1ccf8dd420f826687f6a7242b03"
format_stamp: "Formatted at 2026-08-12 00:14:17 on dist-test-slave-zr1t"
server_key: "ad1c261d4937501336c9a046cd6521df"
server_key_iv: "7053f05af4021070f0ae5501d3be206d"
server_key_version: "encryptionkey@0"
I20260812 00:14:17.833847 23707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:17.839149 23697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } attempt: 2
I20260812 00:14:17.839536 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.839397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55128)
0812 00:14:17.839459 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.839503 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:14:17.839881 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.839799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55140)
0812 00:14:17.839842 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.839865 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 00:14:17.839998 23697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:39739: Network error: Client connection negotiation failed: client connection to 127.31.227.60:39739: connect: Connection refused (error 111)
I20260812 00:14:17.856072 23707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:17.858577 23707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:17.858742 23707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:17.863085 23707 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.60:39739
I20260812 00:14:17.863125 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.60:39739 every 8 connection(s)
I20260812 00:14:17.863535 23707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
I20260812 00:14:17.864423 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:17.865713 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 23707
I20260812 00:14:17.865837 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493496733958-32652-0/minicluster-data/master-2/wal/instance
I20260812 00:14:17.866048 32652 external_mini_cluster.cc:1428] Setting key 87360c37631d7a391ce38a6ce74f0bf5
I20260812 00:14:17.866268 23808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:14:17.875427 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.867115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.31.227.62:44625)
0812 00:14:17.867168 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.867170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.867315 (+ 145us) server_negotiation.cc:407] Connection header received
0812 00:14:17.867479 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.867480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.867493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.867554 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.868150 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.868652 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.869632 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.869828 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.869915 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.872315 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.872317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.872318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.872326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.873844 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.874316 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.874317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.874317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.874378 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.874676 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.874677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.874677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.874805 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.874887 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.875213 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.875302 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 00:14:17.875783 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.867037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:44625 (local address 127.0.0.1:59040)
0812 00:14:17.867180 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.867223 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.867383 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.867576 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.867580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.867871 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.868033 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.868037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.868747 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.868749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.869501 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.869503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.869928 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.869929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.870062 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.870608 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.870619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.872210 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.873938 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.873946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.873948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.874234 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.874463 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.874464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.874464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.874576 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.874877 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.874884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.875060 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.875173 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.875313 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260812 00:14:17.875794 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.866600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.31.227.60:39739)
0812 00:14:17.866773 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.866781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.866807 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:14:17.867369 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.867377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.867398 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.867526 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.868163 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.868979 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.869765 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.870096 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.870188 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.872812 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.872819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.872820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.872826 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.874563 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.874914 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.874918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.874918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.874976 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.875185 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.875186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.875187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.875352 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.875500 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.875629 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.875688 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260812 00:14:17.876402 23808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:17.883409 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.876651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:32829 (local address 127.0.0.1:43932)
0812 00:14:17.876768 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.876779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.876831 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.876952 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.876953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.877089 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.877125 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.877126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.877967 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.877968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.878499 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.878500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.878914 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.878915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.878993 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.879356 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.879360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.880763 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.882403 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.882404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.882405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.882571 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.882780 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.882781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.882781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.882843 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.883098 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.883099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.883140 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.883218 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.883317 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 00:14:17.883417 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.876739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.31.227.61:32829)
0812 00:14:17.876790 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.876792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.876819 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:14:17.876885 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.876886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.876896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.876952 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.877205 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.877890 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.878578 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.878842 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.878899 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.880865 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.880866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.880867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.880874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.882308 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.882643 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.882644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.882645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.882703 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.882911 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.882912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.882913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.883022 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.883093 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.883260 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.883333 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:14:17.883715 23808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:17.884354 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.876726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.31.227.62:44625)
0812 00:14:17.876763 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.876765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.876773 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:14:17.876798 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.876799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.876808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.876872 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.877476 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.878033 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.878730 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.878933 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.878996 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.881504 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.881506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.881506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.881513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.883035 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.883397 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.883399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.883399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.883482 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.883710 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.883711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.883711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.883829 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.883998 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.884232 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.884297 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:14:17.891731 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.883973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55154)
0812 00:14:17.884044 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.884053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.884093 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.884893 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.884894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.885058 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.885084 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.885085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.885887 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.885889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.886619 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.886621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.887510 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.887511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.887557 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.887959 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.887963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.889239 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.890755 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.890756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.890757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.890913 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.891119 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.891120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.891121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.891175 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.891416 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.891416 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.891453 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.891523 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.891618 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 00:14:17.891788 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.884014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.31.227.60:39739)
0812 00:14:17.884056 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.884058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.884087 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:14:17.884144 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.884145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.884157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.884207 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.885170 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.885812 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.887222 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.887433 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.887500 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.889322 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.889323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.889323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.889330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.890662 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.890997 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.890998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.890998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.891053 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.891245 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.891246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.891246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.891350 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.891431 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.891545 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.891667 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":20}
I20260812 00:14:17.892760 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.885329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.31.227.61:32829)
0812 00:14:17.885363 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.885365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.885385 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:14:17.885406 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.885407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.885415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.885470 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.885927 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.886611 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.887210 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.887379 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.887437 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.889990 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.889992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.889992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.889999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.891416 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.891967 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.891968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.891968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.892030 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.892305 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.892306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.892306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.892410 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.892555 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.892646 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.892704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:14:17.893015 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } attempt: 2
I20260812 00:14:17.893608 23808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: Bootstrap starting.
I20260812 00:14:17.894416 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:17.894723 23808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:17.895027 23808 log.cc:824] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:17.895833 23808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: No bootstrap required, opened a new log
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:14:17.898214 23808 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:17.898401 23808 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:17.898440 23808 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1fd1ccf8dd420f826687f6a7242b03, State: Initialized, Role: FOLLOWER
I20260812 00:14:17.898593 23808 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:17.898926 23808 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 00:14:17.901033 23846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:17.901096 23846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:17.901450 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:17.901566 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:17.901744 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.893290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55156)
0812 00:14:17.893393 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.893405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.893454 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.893651 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.893652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.893817 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.893861 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.893863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.894816 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.894818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.895610 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.895613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.896106 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.896107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.896169 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.896593 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.896599 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.898052 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.900305 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.900307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.900307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.900554 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.901037 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.901038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.901039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.901102 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.901446 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.901448 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.901493 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.901575 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.901673 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 00:14:17.902010 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.893333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.31.227.60:39739)
0812 00:14:17.893365 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.893367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.893445 (+ 78us) server_negotiation.cc:407] Connection header received
0812 00:14:17.893510 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.893510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.893588 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.893644 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.894063 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.894731 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.895700 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.895979 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.896049 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.898151 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.898152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.898152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.898159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.900205 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.900902 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.900903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.900903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.900962 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.901176 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.901177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.901178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.901345 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.901416 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.901883 (+ 467us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.901961 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 00:14:17.903303 23571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: Bootstrap starting.
I20260812 00:14:17.903956 23571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:17.904276 23571 log.cc:824] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:17.901254 23707 master_runner.cc:186] Error getting master registration for 127.31.227.62:44625: OK, instance_id { permanent_uuid: "748e4f00947d47bfb731938bba209b94" instance_seqno: 1786493657550152 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:14:17.904505 23707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:17.905112 23571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: No bootstrap required, opened a new log
I20260812 00:14:17.907691 23571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:17.907859 23571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:17.907908 23571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748e4f00947d47bfb731938bba209b94, State: Initialized, Role: FOLLOWER
I20260812 00:14:17.908082 23571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:17.908370 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:17.908425 23571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:17.908897 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.896037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.31.227.61:32829)
0812 00:14:17.896074 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.896076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.896290 (+ 214us) server_negotiation.cc:407] Connection header received
0812 00:14:17.896459 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.896459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.896470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.896502 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.900006 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.900676 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.902396 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.902743 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.902796 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.905137 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.905139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.905139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.905146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.907019 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.907413 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.907414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.907414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.907471 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.907691 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.907692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.907692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.907826 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.908507 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.908719 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.908803 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:14:17.908967 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.896234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.31.227.60:39739)
0812 00:14:17.896521 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.896523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.896674 (+ 151us) server_negotiation.cc:407] Connection header received
0812 00:14:17.896695 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.896696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.896706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.896772 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.897419 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.898211 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.899167 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.899495 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.899554 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.905716 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.905717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.905718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.905725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.907413 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.907809 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.907810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.907810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.907864 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.908071 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.908072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.908072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.908215 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.908363 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.908789 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.908864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":73,"threads_started":1}
I20260812 00:14:17.909248 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.895854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.31.227.62:44625)
0812 00:14:17.895961 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.895963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.896193 (+ 230us) server_negotiation.cc:407] Connection header received
0812 00:14:17.897302 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.897304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.897317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.897350 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.898169 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.898853 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.899971 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.900237 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.900295 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.904601 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.904603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.904603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.904611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.907185 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.907620 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.907621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.907621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.907701 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.908081 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.908082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.908082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.908588 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.908697 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.909075 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.909147 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 00:14:17.910434 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:14:17.911000 23868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:17.911059 23868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:17.911715 23463 master_runner.cc:186] Error getting master registration for 127.31.227.61:32829: OK, instance_id { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" instance_seqno: 1786493657696933 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:14:17.913465 23463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:17.931195 23697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } attempt: 3
Aug 12 00:14:17 dist-test-slave-zr1t krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493657, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:14:17.939096 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.931604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.31.227.60:39739)
0812 00:14:17.931647 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:17.931649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:17.931679 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:14:17.931759 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:17.931760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:17.931771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:17.931854 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:17.932174 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.932719 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.933463 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.933772 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.933855 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.936108 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:17.936110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:17.936110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:17.936119 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:17.937896 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.938248 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.938250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.938250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.938313 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:17.938529 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:17.938530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:17.938531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:17.938655 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:17.938729 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:17.938916 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 00:14:17.939010 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:17.939121 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:17.931568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:39739 (local address 127.0.0.1:55164)
0812 00:14:17.931627 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:17.931640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:17.931695 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:17.931861 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:17.931862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:17.932043 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:17.932090 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.932091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.932809 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.932811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.933375 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:17.933377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:17.933852 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:17.933854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:17.933912 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:17.934619 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:17.934625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:17.935996 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:17.937982 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.937984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.937984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.938171 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.938390 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:17.938392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:17.938392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:17.938457 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:17.938732 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:17.938734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:17.938783 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:14:17.938869 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:14:17.938992 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:14:17.940924 23697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b: Bootstrap starting.
I20260812 00:14:17.941687 23697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:17.942059 23697 log.cc:824] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:17.942926 23697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b: No bootstrap required, opened a new log
I20260812 00:14:17.944613 23697 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:17.944759 23697 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:17.944808 23697 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8db24742fd744fca990025a8b08c87b, State: Initialized, Role: FOLLOWER
I20260812 00:14:17.944940 23697 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:17.945233 23872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:17.945374 23872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:17.945307 23697 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: configured and running, proceeding with master startup.
W20260812 00:14:17.947634 23893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:17.947747 23893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:17.949975 23584 master_runner.cc:263] No leader master found from master e8db24742fd744fca990025a8b08c87b
I20260812 00:14:17.950032 23584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:18.032470 23872 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:18.032616 23872 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:18.032905 23872 leader_election.cc:288] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 748e4f00947d47bfb731938bba209b94 (127.31.227.62:44625), aa1fd1ccf8dd420f826687f6a7242b03 (127.31.227.60:39739)
I20260812 00:14:18.033314 23771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8db24742fd744fca990025a8b08c87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" is_pre_election: true
I20260812 00:14:18.033497 23771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8db24742fd744fca990025a8b08c87b in term 0.
I20260812 00:14:18.033797 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8db24742fd744fca990025a8b08c87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748e4f00947d47bfb731938bba209b94" is_pre_election: true
I20260812 00:14:18.033970 23527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8db24742fd744fca990025a8b08c87b in term 0.
I20260812 00:14:18.034201 23612 leader_election.cc:302] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748e4f00947d47bfb731938bba209b94, e8db24742fd744fca990025a8b08c87b; no voters:
I20260812 00:14:18.034350 23872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:14:18.034406 23872 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:14:18.034438 23872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:18.035461 23872 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:18.035602 23872 leader_election.cc:288] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [CANDIDATE]: Term 1 election: Requested vote from peers 748e4f00947d47bfb731938bba209b94 (127.31.227.62:44625), aa1fd1ccf8dd420f826687f6a7242b03 (127.31.227.60:39739)
I20260812 00:14:18.035938 23771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8db24742fd744fca990025a8b08c87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1fd1ccf8dd420f826687f6a7242b03"
I20260812 00:14:18.036046 23771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:18.037006 23771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8db24742fd744fca990025a8b08c87b in term 1.
I20260812 00:14:18.037246 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8db24742fd744fca990025a8b08c87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748e4f00947d47bfb731938bba209b94"
I20260812 00:14:18.037343 23527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:18.037640 23610 leader_election.cc:302] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1fd1ccf8dd420f826687f6a7242b03, e8db24742fd744fca990025a8b08c87b; no voters:
I20260812 00:14:18.038506 23527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8db24742fd744fca990025a8b08c87b in term 1.
I20260812 00:14:18.038707 23872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:18.038884 23872 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [term 1 LEADER]: Becoming Leader. State: Replica: e8db24742fd744fca990025a8b08c87b, State: Running, Role: LEADER
I20260812 00:14:18.038991 23872 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } }
I20260812 00:14:18.039413 23872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8db24742fd744fca990025a8b08c87b. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:18.039499 23872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:18.040628 23901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:18.040846 23901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:18.043099 23771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [term 1 FOLLOWER]: Refusing update from remote peer e8db24742fd744fca990025a8b08c87b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:18.043375 23527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [term 1 FOLLOWER]: Refusing update from remote peer e8db24742fd744fca990025a8b08c87b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:18.043742 23872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [LEADER]: Connected to new peer: Peer: permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:18.043915 23872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:18.045192 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8db24742fd744fca990025a8b08c87b. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:18.045306 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:18.046352 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8db24742fd744fca990025a8b08c87b. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:18.046459 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:18.047917 23872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
W20260812 00:14:18.047962 23497 tablet.cc:2369] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:14:18.048005 23872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:18.048415 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:18.048501 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:18.048707 23902 mvcc.cc:205] Tried to move back new op lower bound from 7317478023341563904 to 7317478023328374784. Current Snapshot: MvccSnapshot[applied={T|T < 7317478023341563904}]
I20260812 00:14:18.048897 23907 mvcc.cc:205] Tried to move back new op lower bound from 7317478023341563904 to 7317478023328374784. Current Snapshot: MvccSnapshot[applied={T|T < 7317478023341563904}]
I20260812 00:14:18.049692 23872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:18.049778 23872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:18.050135 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8db24742fd744fca990025a8b08c87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748e4f00947d47bfb731938bba209b94" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 44625 } } peers { permanent_uuid: "e8db24742fd744fca990025a8b08c87b" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 32829 } } peers { permanent_uuid: "aa1fd1ccf8dd420f826687f6a7242b03" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 39739 } } }
I20260812 00:14:18.050221 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:18.050886 23905 mvcc.cc:205] Tried to move back new op lower bound from 7317478023341563904 to 7317478023328374784. Current Snapshot: MvccSnapshot[applied={T|T < 7317478023341563904}]
I20260812 00:14:18.051855 23901 catalog_manager.cc:1347] Generated new cluster ID: c3f6ae07b511424b80073c717cef2c11
I20260812 00:14:18.051921 23901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:18.059902 23901 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:18.060635 23901 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:18.070609 23901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8db24742fd744fca990025a8b08c87b: Generated new TSK 0
I20260812 00:14:18.070775 23901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:18.901585 23846 catalog_manager.cc:1259] Loaded cluster ID: c3f6ae07b511424b80073c717cef2c11
I20260812 00:14:18.901688 23846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: loading cluster ID for follower catalog manager: success
I20260812 00:14:18.902575 23846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: acquiring CA information for follower catalog manager: success
I20260812 00:14:18.902905 23846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa1fd1ccf8dd420f826687f6a7242b03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:14:18.911471 23868 catalog_manager.cc:1259] Loaded cluster ID: c3f6ae07b511424b80073c717cef2c11
I20260812 00:14:18.911530 23868 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: loading cluster ID for follower catalog manager: success
I20260812 00:14:18.912406 23868 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: acquiring CA information for follower catalog manager: success
I20260812 00:14:18.912726 23868 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 748e4f00947d47bfb731938bba209b94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:14:19.123726 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 23463
I20260812 00:14:19.138504 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 23584
I20260812 00:14:19.147986 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 23707
2026-08-12T00:14:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3134 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:19 dist-test-slave-zr1t krb5kdc[23931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:19 dist-test-slave-zr1t krb5kdc[23931](info): set up 2 sockets
Aug 12 00:14:19 dist-test-slave-zr1t krb5kdc[23931](info): commencing operation
krb5kdc: starting...
W20260812 00:14:20.392128 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:14:20 dist-test-slave-zr1t krb5kdc[23931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:14:20.508667 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42757
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:20.590323 23945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:20.590497 23945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:20.590516 23945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:20.591671 23945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:20.591712 23945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:20.591727 23945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:20.591739 23945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:20.591789 23945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:20.593209 23945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:20.593447 23945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:20.593716 23945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:14:20.598593 23945 server_base.cc:1034] running on GCE node
W20260812 00:14:20.598567 23950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:20.598558 23955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:20.598567 23951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:20.599077 23945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:20.599421 23945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:20.600584 23945 hybrid_clock.cc:648] HybridClock initialized: now 1786493660600581 us; error 35 us; skew 500 ppm
Aug 12 00:14:20 dist-test-slave-zr1t krb5kdc[23931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493660, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:20.603087 23945 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:20.603525 23945 webserver.cc:460] Webserver started at http://127.31.227.62:34039/ using document root <none> and password file <none>
I20260812 00:14:20.603744 23945 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:20.603809 23945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:20.603964 23945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:20.605018 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "d42bc91a88b048aaba65fd412660cf36"
format_stamp: "Formatted at 2026-08-12 00:14:20 on dist-test-slave-zr1t"
server_key: "05b60222e1c25308a8583e7666e07fce"
server_key_iv: "47b005943d2e7d3de341ef7d655bc7a1"
server_key_version: "encryptionkey@0"
I20260812 00:14:20.605391 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "d42bc91a88b048aaba65fd412660cf36"
format_stamp: "Formatted at 2026-08-12 00:14:20 on dist-test-slave-zr1t"
server_key: "05b60222e1c25308a8583e7666e07fce"
server_key_iv: "47b005943d2e7d3de341ef7d655bc7a1"
server_key_version: "encryptionkey@0"
I20260812 00:14:20.606752 23945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:20.607518 23969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:20.607708 23945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:20.607817 23945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "d42bc91a88b048aaba65fd412660cf36"
format_stamp: "Formatted at 2026-08-12 00:14:20 on dist-test-slave-zr1t"
server_key: "05b60222e1c25308a8583e7666e07fce"
server_key_iv: "47b005943d2e7d3de341ef7d655bc7a1"
server_key_version: "encryptionkey@0"
I20260812 00:14:20.607892 23945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:20.618024 23945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:20.619678 23945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:20.619831 23945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:20.623699 23945 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42757
I20260812 00:14:20.623733 24053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42757 every 8 connection(s)
I20260812 00:14:20.624064 23945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:20.624897 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:20.627203 24054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36: Bootstrap starting.
I20260812 00:14:20.627790 24054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:20.628073 24054 log.cc:824] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:20.628778 24054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36: No bootstrap required, opened a new log
I20260812 00:14:20.630424 24054 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42bc91a88b048aaba65fd412660cf36" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42757 } }
I20260812 00:14:20.630553 24054 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:20.630597 24054 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42bc91a88b048aaba65fd412660cf36, State: Initialized, Role: FOLLOWER
I20260812 00:14:20.630757 24054 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42bc91a88b048aaba65fd412660cf36" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42757 } }
I20260812 00:14:20.630841 24054 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:20.630889 24054 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:20.630937 24054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:20.631438 24054 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42bc91a88b048aaba65fd412660cf36" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42757 } }
I20260812 00:14:20.631551 24054 leader_election.cc:302] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42bc91a88b048aaba65fd412660cf36; no voters:
I20260812 00:14:20.631734 24054 leader_election.cc:288] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:20.631789 24059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:20.631942 24059 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [term 1 LEADER]: Becoming Leader. State: Replica: d42bc91a88b048aaba65fd412660cf36, State: Running, Role: LEADER
I20260812 00:14:20.631970 24054 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:20.632027 24059 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42bc91a88b048aaba65fd412660cf36" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42757 } }
I20260812 00:14:20.632381 24062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42bc91a88b048aaba65fd412660cf36. Latest consensus state: current_term: 1 leader_uuid: "d42bc91a88b048aaba65fd412660cf36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42bc91a88b048aaba65fd412660cf36" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42757 } } }
I20260812 00:14:20.632469 24062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:20.632583 24061 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42bc91a88b048aaba65fd412660cf36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42bc91a88b048aaba65fd412660cf36" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42757 } } }
I20260812 00:14:20.632678 24061 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:20.632720 24067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:20.633260 24067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:20.633711 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 23945
I20260812 00:14:20.633834 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:14:20.634074 32652 external_mini_cluster.cc:1428] Setting key 2f9c2808cbe879228272145c4cca55e4
I20260812 00:14:20.635074 24067 catalog_manager.cc:1347] Generated new cluster ID: 83686c7784054ef59e8d9313c0211ea7
I20260812 00:14:20.635128 24067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:14:20 dist-test-slave-zr1t krb5kdc[23931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:20.643754 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:20.634475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.31.227.62:42757)
0812 00:14:20.634662 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:20.634671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:20.634708 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:14:20.634869 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:20.634876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:20.634918 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:20.635070 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:20.635790 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:20.636680 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:20.637441 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:20.637930 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:20.638054 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:20.640285 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:20.640289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:20.640289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:20.640300 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:20.642127 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:20.642545 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:20.642549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:20.642549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:20.642650 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:20.642879 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:20.642880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:20.642880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:20.643111 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:20.643278 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:20.643402 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 00:14:20.643583 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260812 00:14:20.644747 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:20.647891 24067 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:20.648551 24067 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:20.663137 24067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42bc91a88b048aaba65fd412660cf36: Generated new TSK 0
I20260812 00:14:20.663328 24067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:20.698671 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 23945
2026-08-12T00:14:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1551 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:20 dist-test-slave-zr1t krb5kdc[24089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:20 dist-test-slave-zr1t krb5kdc[24089](info): set up 2 sockets
Aug 12 00:14:20 dist-test-slave-zr1t krb5kdc[24089](info): commencing operation
krb5kdc: starting...
W20260812 00:14:21.963325 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:14:22.089258 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34739
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:22.173379 24105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:22.173729 24105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:22.173772 24105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:22.174975 24105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:22.175064 24105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:22.175089 24105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:22.175113 24105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:22.175140 24105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:22.176676 24105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:22.176985 24105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:22.177230 24105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:22.182579 24110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.182574 24113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.182570 24111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:22.182785 24105 server_base.cc:1034] running on GCE node
I20260812 00:14:22.183032 24105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:22.183342 24105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:22.184528 24105 hybrid_clock.cc:648] HybridClock initialized: now 1786493662184505 us; error 41 us; skew 500 ppm
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:22.187145 24105 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:22.187667 24105 webserver.cc:460] Webserver started at http://127.31.227.62:46075/ using document root <none> and password file <none>
I20260812 00:14:22.187917 24105 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:22.187988 24105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:22.188146 24105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:22.189281 24105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "1208c5b421e24a91be3b03f08461ab3f"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c104b11b53ded2a702782ac5490737e4"
server_key_iv: "cf3be33b22c905e8e49cb7427257e70f"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.189697 24105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "1208c5b421e24a91be3b03f08461ab3f"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c104b11b53ded2a702782ac5490737e4"
server_key_iv: "cf3be33b22c905e8e49cb7427257e70f"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.191071 24105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:22.191824 24124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.192000 24105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.192111 24105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "1208c5b421e24a91be3b03f08461ab3f"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c104b11b53ded2a702782ac5490737e4"
server_key_iv: "cf3be33b22c905e8e49cb7427257e70f"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.192188 24105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:22.201329 24105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:22.203804 24105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:22.203971 24105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:22.208292 24105 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34739
I20260812 00:14:22.208328 24211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34739 every 8 connection(s)
I20260812 00:14:22.208745 24105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:22.209789 24212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:22.212098 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap starting.
I20260812 00:14:22.212726 24212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:22.213024 24212 log.cc:824] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:22.213861 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: No bootstrap required, opened a new log
I20260812 00:14:22.214759 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 24105
I20260812 00:14:22.214877 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:14:22.215108 32652 external_mini_cluster.cc:1428] Setting key eb2e9b3179f4f88d285200ef632d1dce
I20260812 00:14:22.216097 24212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:22.216262 24212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:22.216327 24212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1208c5b421e24a91be3b03f08461ab3f, State: Initialized, Role: FOLLOWER
I20260812 00:14:22.216518 24212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:22.216619 24212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:22.216663 24212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:22.216714 24212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:22.217646 24212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:22.217763 24212 leader_election.cc:302] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1208c5b421e24a91be3b03f08461ab3f; no voters:
I20260812 00:14:22.217965 24212 leader_election.cc:288] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:22.218015 24219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:22.218150 24219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 LEADER]: Becoming Leader. State: Replica: 1208c5b421e24a91be3b03f08461ab3f, State: Running, Role: LEADER
I20260812 00:14:22.218233 24212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:22.218293 24219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:22.218715 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1208c5b421e24a91be3b03f08461ab3f. Latest consensus state: current_term: 1 leader_uuid: "1208c5b421e24a91be3b03f08461ab3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } } }
I20260812 00:14:22.218861 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:22.218863 24223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1208c5b421e24a91be3b03f08461ab3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } } }
I20260812 00:14:22.219025 24223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: This master's current role is: LEADER
W20260812 00:14:22.220585 24243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:22.220630 24243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:22.220700 24229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:22.220861 24229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:22.222399 24229 catalog_manager.cc:1347] Generated new cluster ID: b2a1d0815c6542d8b7619a6967246f3a
I20260812 00:14:22.222457 24229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:22.226853 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.215689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.31.227.62:34739)
0812 00:14:22.215877 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.215888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.215931 (+ 43us) server_negotiation.cc:407] Connection header received
0812 00:14:22.216054 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.216061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.216104 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.216239 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.217032 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.218312 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.219496 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.220001 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.220102 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.223067 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.223071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.223071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:22.223085 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.224951 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.225324 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.225327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.225327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.225460 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.225691 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.225692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.225693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.225946 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.226115 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.226225 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.226378 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260812 00:14:22.227062 24229 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:22.227854 24229 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:22.234714 24229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Generated new TSK 0
I20260812 00:14:22.234921 24229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:14:22.268088 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34739
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:22.370841 24254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:22.371057 24254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:22.371081 24254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:22.372243 24254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:22.372289 24254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:22.372315 24254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:14:22.373867 24254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:22.374130 24254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:22.374413 24254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:22.380125 24266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.380098 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.380115 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:22.380299 24254 server_base.cc:1034] running on GCE node
I20260812 00:14:22.380614 24254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:22.380887 24254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:22.382030 24254 hybrid_clock.cc:648] HybridClock initialized: now 1786493662382007 us; error 38 us; skew 500 ppm
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:22.384645 24254 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:14:22.385102 24254 webserver.cc:460] Webserver started at http://127.31.227.1:32835/ using document root <none> and password file <none>
I20260812 00:14:22.385329 24254 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:22.385375 24254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:22.385581 24254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:22.386714 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "87cdd2fcf958446db946e853c0fd9b48"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "50a732cbcd71219f1c2cace1f0894595"
server_key_iv: "97ae7d5e4535c22b2e7b817485c49940"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.387104 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "87cdd2fcf958446db946e853c0fd9b48"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "50a732cbcd71219f1c2cace1f0894595"
server_key_iv: "97ae7d5e4535c22b2e7b817485c49940"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.388553 24254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:22.389329 24276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.389572 24254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:22.389665 24254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "87cdd2fcf958446db946e853c0fd9b48"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "50a732cbcd71219f1c2cace1f0894595"
server_key_iv: "97ae7d5e4535c22b2e7b817485c49940"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.389744 24254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:22.417766 24254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:22.420295 24254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:22.420428 24254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:22.420679 24254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:22.421092 24254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:22.421128 24254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.421178 24254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:22.421205 24254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.427668 24254 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:36041
I20260812 00:14:22.427754 24455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:36041 every 8 connection(s)
I20260812 00:14:22.428104 24254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:22.435853 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 24254
I20260812 00:14:22.435977 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:14:22.436164 32652 external_mini_cluster.cc:1428] Setting key 7a8d18e1e75b0bb5360686cbdaa36fbf
I20260812 00:14:22.439216 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.428881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:51211 (local address 127.31.227.62:34739)
0812 00:14:22.428930 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.428932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.429188 (+ 256us) server_negotiation.cc:407] Connection header received
0812 00:14:22.429820 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.429821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.429838 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.429892 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.430916 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.431451 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.432482 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.432715 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.432805 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.435487 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.435490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.435490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:22.435503 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.437387 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.437977 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.437978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.437979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.438042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.438367 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.438369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.438369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.438538 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.438619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.438998 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.439112 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:22.439867 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.428869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:51211)
0812 00:14:22.429063 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.429102 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.429709 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.429981 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.429985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.430588 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:22.430792 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.430796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.431557 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.431558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.432384 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.432387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.432820 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.432821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.432950 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.433461 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:22.433470 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:22.435357 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:22.437485 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:22.437493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:22.437495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:22.437865 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:22.438133 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:22.438134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:22.438134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:22.438278 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:22.438619 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:22.438626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:22.438807 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.439323 (+ 516us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.439567 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260812 00:14:22.440361 24456 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34739
I20260812 00:14:22.440496 24456 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:22.440788 24456 heartbeater.cc:511] Master 127.31.227.62:34739 requested a full tablet report, sending...
I20260812 00:14:22.441385 24155 ts_manager.cc:194] Registered new tserver with Master: 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041)
I20260812 00:14:22.442458 24155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:51211
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:14:22.468050 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34739
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:22.552482 24468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:22.552672 24468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:22.552693 24468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:22.553794 24468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:22.553830 24468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:22.553855 24468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:14:22.555238 24468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:22.555429 24468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:22.555598 24468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:22.560856 24473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.560837 24474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.560832 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:22.561034 24468 server_base.cc:1034] running on GCE node
I20260812 00:14:22.561300 24468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:22.561614 24468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:22.562784 24468 hybrid_clock.cc:648] HybridClock initialized: now 1786493662562769 us; error 45 us; skew 500 ppm
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:22.569749 24468 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:14:22.570250 24468 webserver.cc:460] Webserver started at http://127.31.227.2:43249/ using document root <none> and password file <none>
I20260812 00:14:22.570501 24468 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:22.570544 24468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:22.570632 24468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:22.571637 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "3cbee32b549b45e0bc9d662cf4d538ca"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c20ee65f029109323d466679e47f77f2"
server_key_iv: "8273c3006bfb7492dac63674c9ba23e8"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.571959 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "3cbee32b549b45e0bc9d662cf4d538ca"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c20ee65f029109323d466679e47f77f2"
server_key_iv: "8273c3006bfb7492dac63674c9ba23e8"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.573231 24468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:14:22.574038 24489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.574234 24468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:22.574332 24468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "3cbee32b549b45e0bc9d662cf4d538ca"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c20ee65f029109323d466679e47f77f2"
server_key_iv: "8273c3006bfb7492dac63674c9ba23e8"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.574416 24468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:22.590958 24468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:22.593132 24468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:22.593292 24468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:22.593573 24468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:22.593978 24468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:22.594036 24468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.594074 24468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:22.594122 24468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.600734 24468 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:32939
I20260812 00:14:22.600780 24668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:32939 every 8 connection(s)
I20260812 00:14:22.601106 24468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:14:22.603936 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 24468
I20260812 00:14:22.604071 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:14:22.604275 32652 external_mini_cluster.cc:1428] Setting key e824cc7528bb2318176c4c53ce555dd8
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:22.611573 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.601839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:49141 (local address 127.31.227.62:34739)
0812 00:14:22.601895 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.601897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.602133 (+ 236us) server_negotiation.cc:407] Connection header received
0812 00:14:22.602713 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.602715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.602729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.602782 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.603770 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.604456 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.605647 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.605853 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.605943 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.608333 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.608334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.608335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:22.608345 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.609996 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.610472 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.610473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.610474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.610558 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.610809 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.610810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.610810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.610978 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.611059 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.611334 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.611431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:14:22.611594 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.601837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:49141)
0812 00:14:22.601997 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.602036 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.602583 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.602859 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.602863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.603454 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:22.603658 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.603663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.604554 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.604556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.605559 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.605564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.605955 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.605957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.606085 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.606641 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:22.606651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:22.608235 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:22.610074 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:22.610081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:22.610083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:22.610391 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:22.610632 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:22.610633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:22.610633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:22.610737 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:22.611052 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:22.611058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:22.611190 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.611280 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.611424 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260812 00:14:22.612169 24669 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34739
I20260812 00:14:22.612289 24669 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:22.612511 24669 heartbeater.cc:511] Master 127.31.227.62:34739 requested a full tablet report, sending...
I20260812 00:14:22.612972 24155 ts_manager.cc:194] Registered new tserver with Master: 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939)
I20260812 00:14:22.613366 24155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:49141
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:14:22.632532 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34739
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:22.716948 24677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:22.717180 24677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:22.717222 24677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:22.718518 24677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:22.718583 24677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:22.718631 24677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:14:22.720254 24677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:22.720539 24677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:22.720788 24677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:22.726011 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.726003 24685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.725986 24686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:22.726462 24677 server_base.cc:1034] running on GCE node
I20260812 00:14:22.726732 24677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:22.726981 24677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:22.728120 24677 hybrid_clock.cc:648] HybridClock initialized: now 1786493662728108 us; error 28 us; skew 500 ppm
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:22.730757 24677 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:14:22.731196 24677 webserver.cc:460] Webserver started at http://127.31.227.3:46421/ using document root <none> and password file <none>
I20260812 00:14:22.731439 24677 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:22.731510 24677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:22.731665 24677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:22.732769 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "5931c9c1a4595232eac37d7a7065979e"
server_key_iv: "6159a308696ae7475f1c1c320c45f4d6"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.733147 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "5931c9c1a4595232eac37d7a7065979e"
server_key_iv: "6159a308696ae7475f1c1c320c45f4d6"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.734680 24677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:14:22.735502 24700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.735675 24677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:14:22.735786 24677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "5931c9c1a4595232eac37d7a7065979e"
server_key_iv: "6159a308696ae7475f1c1c320c45f4d6"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.735869 24677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:22.754208 24677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:22.756098 24677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:22.756275 24677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:22.756538 24677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:22.756858 24677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:22.756912 24677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.756953 24677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:22.757014 24677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.763324 24677 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:38511
I20260812 00:14:22.763376 24867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:38511 every 8 connection(s)
I20260812 00:14:22.763715 24677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:14:22.768491 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 24677
I20260812 00:14:22.768596 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:14:22.768787 32652 external_mini_cluster.cc:1428] Setting key 731be3eb8e737818c0e957505a4fbdb4
Aug 12 00:14:22 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:22.773270 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.764470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:57845 (local address 127.31.227.62:34739)
0812 00:14:22.764545 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.764547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.764780 (+ 233us) server_negotiation.cc:407] Connection header received
0812 00:14:22.765333 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.765335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.765349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.765402 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.766438 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.766940 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.767960 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.768164 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.768237 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.770616 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.770618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.770618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:22.770626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.771863 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.772281 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.772283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.772283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.772348 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.772609 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.772610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.772611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.772733 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.772824 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.773100 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.773181 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 00:14:22.773465 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.764458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:57845)
0812 00:14:22.764653 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.764691 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.765222 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.765491 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.765497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.766114 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:22.766311 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.766316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.767048 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.767049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.767856 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.767858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.768259 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.768260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.768394 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.768930 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:22.768941 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:22.770521 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:22.771959 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:22.771967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:22.771969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:22.772191 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:22.772430 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:22.772431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:22.772431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:22.772539 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:22.772816 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:22.772824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:22.772962 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.773066 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.773226 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260812 00:14:22.773986 24868 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34739
I20260812 00:14:22.774072 24868 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:22.774226 24868 heartbeater.cc:511] Master 127.31.227.62:34739 requested a full tablet report, sending...
I20260812 00:14:22.774559 24155 ts_manager.cc:194] Registered new tserver with Master: 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
I20260812 00:14:22.774946 24155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:57845
I20260812 00:14:22.780642 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:14:22.788480 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.782113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.31.227.62:34739)
0812 00:14:22.782185 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.782187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.782313 (+ 126us) server_negotiation.cc:407] Connection header received
0812 00:14:22.782420 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.782421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.782432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.782483 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.783071 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.783568 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.784355 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.784568 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.784656 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.785884 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.785885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.785886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:22.785893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.787048 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.787414 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.787416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.787416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.787478 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:22.787721 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:22.787722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:22.787722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:22.787916 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.788036 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.788216 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.788350 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:14:22.790107 24155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:14:22.792233 24155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:14:22.799350 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.795370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.31.227.1:36041)
0812 00:14:22.795616 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.795621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.795631 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:22.795678 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.795680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.795695 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.795764 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:22.795935 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.797140 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.798423 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.798944 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.799033 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.799099 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.799230 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.799278 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260812 00:14:22.799332 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.794956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:36041 (local address 127.0.0.1:33102)
0812 00:14:22.795112 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.795170 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.795238 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.795780 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.795782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.795785 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:22.795846 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.795849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.797250 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.797251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.798316 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.798318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.799033 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.799034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.799116 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.799129 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.799202 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.799258 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260812 00:14:22.799535 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.795094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.31.227.3:38511)
0812 00:14:22.795318 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.795323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.795450 (+ 127us) server_negotiation.cc:407] Connection header received
0812 00:14:22.795504 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.795507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.795526 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.795627 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:22.795932 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.796779 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.798222 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.798975 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.799043 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.799099 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.799384 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.799434 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260812 00:14:22.799605 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.794964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:38511 (local address 127.0.0.1:39906)
0812 00:14:22.795371 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.795380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.795433 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.795755 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.795758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.795761 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:22.795836 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.795839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.796885 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.796886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.798129 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.798132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.799137 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.799139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.799201 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.799206 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.799256 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.799321 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":93,"threads_started":1}
I20260812 00:14:22.800123 24357 tablet_service.cc:1467] Processing CreateTablet for tablet 0a88318ca05546d3b84b92699d36caeb (DEFAULT_TABLE table=test-table [id=f19dc8dbbd90464f8a7f60be4c6ae418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:22.800379 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 0a88318ca05546d3b84b92699d36caeb (DEFAULT_TABLE table=test-table [id=f19dc8dbbd90464f8a7f60be4c6ae418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:22.800530 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.795175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:32939 (local address 127.0.0.1:45868)
0812 00:14:22.795586 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.795615 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.795663 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.796104 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.796105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.796107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:22.796152 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.796154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.797388 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.797389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.798423 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.798426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.800299 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.800300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.800368 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.800373 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.800430 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.800470 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":75,"threads_started":1}
I20260812 00:14:22.800534 24357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a88318ca05546d3b84b92699d36caeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:22.800738 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a88318ca05546d3b84b92699d36caeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:22.800761 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.795354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.31.227.2:32939)
0812 00:14:22.795926 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.795930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.795940 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:22.795981 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.795983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.795999 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.796091 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:22.796204 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.797266 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.798526 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.799270 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.799327 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.799380 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.800628 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.800681 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":113,"threads_started":1}
I20260812 00:14:22.802378 24566 tablet_service.cc:1467] Processing CreateTablet for tablet 0a88318ca05546d3b84b92699d36caeb (DEFAULT_TABLE table=test-table [id=f19dc8dbbd90464f8a7f60be4c6ae418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:22.802726 24566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a88318ca05546d3b84b92699d36caeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:22.803575 24895 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap starting.
I20260812 00:14:22.804051 24896 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Bootstrap starting.
I20260812 00:14:22.804451 24895 tablet_bootstrap.cc:654] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:22.804746 24896 tablet_bootstrap.cc:654] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:22.805095 24895 log.cc:824] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:22.805379 24896 log.cc:824] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:22.805969 24895 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: No bootstrap required, opened a new log
I20260812 00:14:22.806051 24895 ts_tablet_manager.cc:1397] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:14:22.805977 24899 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Bootstrap starting.
I20260812 00:14:22.806166 24896 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: No bootstrap required, opened a new log
I20260812 00:14:22.806246 24896 ts_tablet_manager.cc:1397] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:14:22.806882 24899 tablet_bootstrap.cc:654] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:22.807439 24899 log.cc:824] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:22.807869 24896 raft_consensus.cc:348] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.808053 24896 raft_consensus.cc:374] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:22.808099 24896 raft_consensus.cc:729] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7434a2e78a634cb0a9de3c5a1a75acce, State: Initialized, Role: FOLLOWER
I20260812 00:14:22.808246 24899 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: No bootstrap required, opened a new log
I20260812 00:14:22.808211 24896 consensus_queue.cc:260] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.808156 24895 raft_consensus.cc:348] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.808313 24899 ts_tablet_manager.cc:1397] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:14:22.808314 24895 raft_consensus.cc:374] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:22.808350 24895 raft_consensus.cc:729] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cdd2fcf958446db946e853c0fd9b48, State: Initialized, Role: FOLLOWER
I20260812 00:14:22.808458 24895 consensus_queue.cc:260] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.808557 24896 ts_tablet_manager.cc:1428] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 00:14:22.808684 24868 heartbeater.cc:503] Master 127.31.227.62:34739 was elected leader, sending a full tablet report...
I20260812 00:14:22.808735 24895 ts_tablet_manager.cc:1428] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:14:22.808802 24456 heartbeater.cc:503] Master 127.31.227.62:34739 was elected leader, sending a full tablet report...
I20260812 00:14:22.810655 24899 raft_consensus.cc:348] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.810774 24899 raft_consensus.cc:374] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:22.810796 24899 raft_consensus.cc:729] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbee32b549b45e0bc9d662cf4d538ca, State: Initialized, Role: FOLLOWER
I20260812 00:14:22.810890 24899 consensus_queue.cc:260] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.811189 24669 heartbeater.cc:503] Master 127.31.227.62:34739 was elected leader, sending a full tablet report...
I20260812 00:14:22.811193 24899 ts_tablet_manager.cc:1428] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260812 00:14:22.851770 24671 tablet.cc:2369] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:14:22.895471 24903 raft_consensus.cc:482] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:22.895632 24903 raft_consensus.cc:504] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.896090 24903 leader_election.cc:288] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511), 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939)
I20260812 00:14:22.899098 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.896387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:33167 (local address 127.31.227.3:38511)
0812 00:14:22.896439 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.896441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.896454 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:14:22.896487 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.896488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.896498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.896553 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:22.896794 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.897331 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.898336 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.898721 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.898778 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.898785 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.898983 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.899025 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:14:22.899111 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.896277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:38511 (local address 127.31.227.1:33167)
0812 00:14:22.896343 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.896358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.896428 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.896603 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.896604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.896606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:22.896707 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.896709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.897412 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.897413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.898243 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.898245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.898800 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.898801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.898891 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.898895 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.898954 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.899009 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":43}
I20260812 00:14:22.899135 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.896482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:52251 (local address 127.31.227.2:32939)
0812 00:14:22.896535 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.896538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.896647 (+ 109us) server_negotiation.cc:407] Connection header received
0812 00:14:22.896683 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.896684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.896694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.896753 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:22.896933 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.897382 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.898372 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.898736 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.898825 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.898834 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.899028 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.899071 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:14:22.899132 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.896399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:32939 (local address 127.31.227.1:52251)
0812 00:14:22.896564 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.896574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:22.896641 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:22.896769 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:22.896770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:22.896771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:22.896859 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.896860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.897474 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.897475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.898254 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:22.898257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.898845 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.898846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:22.898921 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.898924 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:14:22.899004 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:14:22.899039 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260812 00:14:22.899557 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a88318ca05546d3b84b92699d36caeb" candidate_uuid: "87cdd2fcf958446db946e853c0fd9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" is_pre_election: true
I20260812 00:14:22.899581 24798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a88318ca05546d3b84b92699d36caeb" candidate_uuid: "87cdd2fcf958446db946e853c0fd9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" is_pre_election: true
I20260812 00:14:22.899701 24595 raft_consensus.cc:2454] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87cdd2fcf958446db946e853c0fd9b48 in term 0.
I20260812 00:14:22.899744 24798 raft_consensus.cc:2454] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87cdd2fcf958446db946e853c0fd9b48 in term 0.
I20260812 00:14:22.899947 24277 leader_election.cc:302] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbee32b549b45e0bc9d662cf4d538ca, 87cdd2fcf958446db946e853c0fd9b48; no voters:
I20260812 00:14:22.900098 24903 raft_consensus.cc:2781] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:14:22.900152 24903 raft_consensus.cc:482] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:14:22.900168 24903 raft_consensus.cc:3037] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:22.900849 24903 raft_consensus.cc:504] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.900974 24903 leader_election.cc:288] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 1 election: Requested vote from peers 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511), 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939)
I20260812 00:14:22.901172 24798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a88318ca05546d3b84b92699d36caeb" candidate_uuid: "87cdd2fcf958446db946e853c0fd9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
I20260812 00:14:22.901196 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a88318ca05546d3b84b92699d36caeb" candidate_uuid: "87cdd2fcf958446db946e853c0fd9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbee32b549b45e0bc9d662cf4d538ca"
I20260812 00:14:22.901280 24595 raft_consensus.cc:3037] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:22.901271 24798 raft_consensus.cc:3037] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:22.902063 24595 raft_consensus.cc:2454] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87cdd2fcf958446db946e853c0fd9b48 in term 1.
I20260812 00:14:22.902107 24798 raft_consensus.cc:2454] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87cdd2fcf958446db946e853c0fd9b48 in term 1.
I20260812 00:14:22.902314 24277 leader_election.cc:302] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7434a2e78a634cb0a9de3c5a1a75acce, 87cdd2fcf958446db946e853c0fd9b48; no voters:
I20260812 00:14:22.902433 24903 raft_consensus.cc:2781] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:22.902619 24903 raft_consensus.cc:686] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 LEADER]: Becoming Leader. State: Replica: 87cdd2fcf958446db946e853c0fd9b48, State: Running, Role: LEADER
I20260812 00:14:22.902732 24903 consensus_queue.cc:237] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:22.903594 24155 catalog_manager.cc:5495] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "87cdd2fcf958446db946e853c0fd9b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } health_report { overall_health: UNKNOWN } } }
I20260812 00:14:22.918045 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.915647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.31.227.1:36041)
0812 00:14:22.915696 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.915699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.915708 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:22.915778 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.915779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.915797 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.915882 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:22.916014 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.916461 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.917198 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.917396 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.917486 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.917655 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:22.917754 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:22.917847 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.917920 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.917937 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:14:22.919888 24595 raft_consensus.cc:1261] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Refusing update from remote peer 87cdd2fcf958446db946e853c0fd9b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:22.919898 24798 raft_consensus.cc:1261] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Refusing update from remote peer 87cdd2fcf958446db946e853c0fd9b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:22.920213 24903 consensus_queue.cc:1035] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:22.920346 24908 consensus_queue.cc:1035] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:22.922848 24915 mvcc.cc:205] Tried to move back new op lower bound from 7317478043317276672 to 7317478043250462720. Current Snapshot: MvccSnapshot[applied={T|T < 7317478043317276672}]
I20260812 00:14:22.923612 24916 mvcc.cc:205] Tried to move back new op lower bound from 7317478043317276672 to 7317478043250462720. Current Snapshot: MvccSnapshot[applied={T|T < 7317478043317276672}]
I20260812 00:14:22.925055 24154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:22.925153 24154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:22.925979 24154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Sending DeleteTablet for 3 replicas of tablet 0a88318ca05546d3b84b92699d36caeb
I20260812 00:14:22.926317 24357 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a88318ca05546d3b84b92699d36caeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:22 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:33102
I20260812 00:14:22.926366 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 24254
I20260812 00:14:22.926540 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a88318ca05546d3b84b92699d36caeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:22 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:39906
I20260812 00:14:22.930842 24566 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a88318ca05546d3b84b92699d36caeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:22 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45868
I20260812 00:14:22.933593 24931 tablet_replica.cc:330] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: stopping tablet replica
I20260812 00:14:22.933732 24931 raft_consensus.cc:2229] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:14:22.933826 24931 raft_consensus.cc:2258] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:22.936213 24931 ts_tablet_manager.cc:1905] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:22.936265 24930 tablet_replica.cc:330] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: stopping tablet replica
I20260812 00:14:22.936379 24930 raft_consensus.cc:2229] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:14:22.936494 24930 raft_consensus.cc:2258] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:22.938862 24931 ts_tablet_manager.cc:1918] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:22.938961 24931 log.cc:1196] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/0a88318ca05546d3b84b92699d36caeb
I20260812 00:14:22.938992 24930 ts_tablet_manager.cc:1905] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:22.939245 24931 ts_tablet_manager.cc:1939] T 0a88318ca05546d3b84b92699d36caeb P 3cbee32b549b45e0bc9d662cf4d538ca: Deleting consensus metadata
I20260812 00:14:22.939747 24126 catalog_manager.cc:4841] TS 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): tablet 0a88318ca05546d3b84b92699d36caeb (table test-table [id=f19dc8dbbd90464f8a7f60be4c6ae418]) successfully deleted
I20260812 00:14:22.940512 24930 ts_tablet_manager.cc:1918] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:22.940573 24930 log.cc:1196] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/0a88318ca05546d3b84b92699d36caeb
I20260812 00:14:22.940781 24930 ts_tablet_manager.cc:1939] T 0a88318ca05546d3b84b92699d36caeb P 7434a2e78a634cb0a9de3c5a1a75acce: Deleting consensus metadata
I20260812 00:14:22.941148 24126 catalog_manager.cc:4841] TS 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): tablet 0a88318ca05546d3b84b92699d36caeb (table test-table [id=f19dc8dbbd90464f8a7f60be4c6ae418]) successfully deleted
W20260812 00:14:22.945602 24128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:36041 (error 108)
I20260812 00:14:22.945763 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 24468
I20260812 00:14:22.950073 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.949914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:36041 (local address 127.0.0.1:33128)
0812 00:14:22.949968 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:22.950027 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260812 00:14:22.953707 24128 catalog_manager.cc:4584] TS 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a88318ca05546d3b84b92699d36caeb: Network error: Client connection negotiation failed: client connection to 127.31.227.1:36041: connect: Connection refused (error 111)
I20260812 00:14:22.954377 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 24677
I20260812 00:14:22.962110 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 24105
I20260812 00:14:22.977406 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34739
--webserver_interface=127.31.227.62
--webserver_port=46075
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:23.063902 24934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.064121 24934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.064143 24934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.065315 24934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:23.065358 24934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.065373 24934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.065385 24934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:23.065434 24934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:23.067095 24934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=46075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:23.067348 24934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.067600 24934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:14:23.078819 24934 server_base.cc:1034] running on GCE node
W20260812 00:14:23.078781 24940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.078794 24942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.078981 24944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.079260 24934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.079625 24934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.080803 24934 hybrid_clock.cc:648] HybridClock initialized: now 1786493663080788 us; error 44 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.083643 24934 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:23.084092 24934 webserver.cc:460] Webserver started at http://127.31.227.62:46075/ using document root <none> and password file <none>
I20260812 00:14:23.084347 24934 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.084446 24934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.085893 24934 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:23.086584 24954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.086753 24934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.086864 24934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "1208c5b421e24a91be3b03f08461ab3f"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c104b11b53ded2a702782ac5490737e4"
server_key_iv: "cf3be33b22c905e8e49cb7427257e70f"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.087281 24934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.110919 24934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:23.113468 24934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:23.113678 24934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:23.117910 24934 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34739
I20260812 00:14:23.117947 25040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34739 every 8 connection(s)
I20260812 00:14:23.118394 24934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:23.120118 25041 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:14:23.120826 25041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap starting.
I20260812 00:14:23.123162 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 24934
I20260812 00:14:23.123528 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:36041
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:34739
--webserver_port=32835
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:23.130986 25041 log.cc:824] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:23.132982 25041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:14:23.133299 25041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap complete.
I20260812 00:14:23.135766 25041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.136018 25041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1208c5b421e24a91be3b03f08461ab3f, State: Initialized, Role: FOLLOWER
I20260812 00:14:23.136224 25041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.136322 25041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:23.136381 25041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:23.136431 25041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:14:23.137454 25041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.137614 25041 leader_election.cc:302] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1208c5b421e24a91be3b03f08461ab3f; no voters:
I20260812 00:14:23.137827 25041 leader_election.cc:288] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:14:23.137866 25049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:14:23.137985 25049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 LEADER]: Becoming Leader. State: Replica: 1208c5b421e24a91be3b03f08461ab3f, State: Running, Role: LEADER
I20260812 00:14:23.138082 25049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.138116 25041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:23.138515 25050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1208c5b421e24a91be3b03f08461ab3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } } }
I20260812 00:14:23.138600 25050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:23.138836 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1208c5b421e24a91be3b03f08461ab3f. Latest consensus state: current_term: 2 leader_uuid: "1208c5b421e24a91be3b03f08461ab3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } } }
I20260812 00:14:23.138916 25058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:23.138936 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:23.139812 25058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:23.140142 25058 catalog_manager.cc:1259] Loaded cluster ID: b2a1d0815c6542d8b7619a6967246f3a
I20260812 00:14:23.140206 25058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:23.140522 25058 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 00:14:23.145606 25058 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 00:14:23.148062 25058 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:23.148267 25058 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Loaded TSK: 0
I20260812 00:14:23.148481 25058 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:14:23.212359 25046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.212555 25046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.212579 25046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.213771 25046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.213810 25046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.213836 25046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:14:23.215317 25046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:36041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=32835
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:23.215554 25046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.215811 25046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:23.220884 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.220922 25087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.221163 25046 server_base.cc:1034] running on GCE node
W20260812 00:14:23.220862 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.221446 25046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.221725 25046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.222886 25046 hybrid_clock.cc:648] HybridClock initialized: now 1786493663222879 us; error 39 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.225611 25046 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:14:23.226109 25046 webserver.cc:460] Webserver started at http://127.31.227.1:32835/ using document root <none> and password file <none>
I20260812 00:14:23.226343 25046 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.226425 25046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.227757 25046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:23.228471 25099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.228618 25046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:14:23.228721 25046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "87cdd2fcf958446db946e853c0fd9b48"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "50a732cbcd71219f1c2cace1f0894595"
server_key_iv: "97ae7d5e4535c22b2e7b817485c49940"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.229067 25046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.246125 25046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:23.248476 25046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:23.248625 25046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:23.248900 25046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:23.249478 25111 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:14:23.251029 25046 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:14:23.251082 25046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:14:23.251116 25046 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:14:23.251796 25046 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:14:23.251842 25046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:23.251890 25111 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap starting.
I20260812 00:14:23.259965 25046 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:36041
I20260812 00:14:23.260285 25289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:36041 every 8 connection(s)
I20260812 00:14:23.260450 25046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:14:23.265143 25111 log.cc:824] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:23.267609 25111 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:14:23.267894 25111 tablet_bootstrap.cc:492] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap complete.
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:23.269166 25111 ts_tablet_manager.cc:1397] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260812 00:14:23.270401 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 25046
I20260812 00:14:23.270737 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:32939
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:34739
--webserver_port=43249
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:23.270565 25111 raft_consensus.cc:348] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:23.270933 25111 raft_consensus.cc:729] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cdd2fcf958446db946e853c0fd9b48, State: Initialized, Role: FOLLOWER
I20260812 00:14:23.271073 25111 consensus_queue.cc:260] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } }
I20260812 00:14:23.271361 25111 ts_tablet_manager.cc:1428] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260812 00:14:23.273676 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.261178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:47057)
0812 00:14:23.261380 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.261429 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.262039 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.262625 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.262629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.262991 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:23.263237 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.263243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.264973 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.264975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.266211 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.266214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.266997 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.266999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.267154 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.267758 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:23.267768 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:23.269679 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:23.272012 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.272019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.272021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.272248 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.272530 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.272531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.272531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.272645 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.273031 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:23.273038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:23.273174 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.273275 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.273454 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
I20260812 00:14:23.273676 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.261324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:47057 (local address 127.31.227.62:34739)
0812 00:14:23.261534 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.261554 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.261596 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:14:23.262309 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.262317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.262405 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.262571 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:23.263388 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.264833 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.266310 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.266863 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.267003 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.269792 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:23.269796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:23.269797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:23.269810 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:23.271911 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.272352 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.272357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.272357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.272448 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.272721 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.272722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.272722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.272948 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:23.273038 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.273355 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.273485 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260812 00:14:23.274387 25290 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34739
I20260812 00:14:23.274489 25290 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:23.274665 25290 heartbeater.cc:511] Master 127.31.227.62:34739 requested a full tablet report, sending...
I20260812 00:14:23.275435 24981 ts_manager.cc:194] Registered new tserver with Master: 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041)
I20260812 00:14:23.276422 24981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:47057
I20260812 00:14:23.277453 25290 heartbeater.cc:503] Master 127.31.227.62:34739 was elected leader, sending a full tablet report...
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.1@KRBTEST.COM
I20260812 00:14:23.284757 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.276111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.31.227.1:36041)
0812 00:14:23.276531 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.276537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.276547 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:23.276582 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.276584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.276599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.276674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:23.277137 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.277998 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.279165 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.279438 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.279493 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.281769 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:23.281776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:23.281777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:23.281783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:23.283404 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.283861 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.283863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.283864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.283923 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.284158 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.284159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.284159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.284311 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:23.284441 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.284616 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.284698 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":84,"threads_started":1}
I20260812 00:14:23.284795 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.276041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:36041 (local address 127.0.0.1:33134)
0812 00:14:23.276282 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.276317 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.276395 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.276692 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.276699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.276999 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:23.277042 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.277044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.278078 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.278079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.279074 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.279078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.279521 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.279523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.279613 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.280034 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:23.280040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:23.281685 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:23.283495 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.283503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.283505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.283769 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.283990 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.283991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.283991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.284062 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.284427 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:23.284433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:23.284488 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.284574 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.284694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260812 00:14:23.285234 25176 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a88318ca05546d3b84b92699d36caeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:22 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:33134
I20260812 00:14:23.285779 25302 tablet_replica.cc:330] stopping tablet replica
I20260812 00:14:23.285887 25302 raft_consensus.cc:2229] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:14:23.285961 25302 raft_consensus.cc:2258] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:23.286262 25302 ts_tablet_manager.cc:1905] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:23.288566 25302 ts_tablet_manager.cc:1918] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:23.288647 25302 log.cc:1196] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/0a88318ca05546d3b84b92699d36caeb
I20260812 00:14:23.288892 25302 ts_tablet_manager.cc:1939] T 0a88318ca05546d3b84b92699d36caeb P 87cdd2fcf958446db946e853c0fd9b48: Deleting consensus metadata
I20260812 00:14:23.289373 24958 catalog_manager.cc:4841] TS 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041): tablet 0a88318ca05546d3b84b92699d36caeb (table test-table [id=f19dc8dbbd90464f8a7f60be4c6ae418]) successfully deleted
W20260812 00:14:23.359297 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.359486 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.359509 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.360574 25299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.360605 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.360630 25299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:14:23.362010 25299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:32939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=43249
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:23.362192 25299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.362372 25299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:23.367427 25313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.367404 25309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.367482 25310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.367679 25299 server_base.cc:1034] running on GCE node
I20260812 00:14:23.367861 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.368052 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.369220 25299 hybrid_clock.cc:648] HybridClock initialized: now 1786493663369200 us; error 30 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.371729 25299 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:14:23.372114 25299 webserver.cc:460] Webserver started at http://127.31.227.2:43249/ using document root <none> and password file <none>
I20260812 00:14:23.372325 25299 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.372378 25299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.373987 25299 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:23.374542 25326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.374696 25299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:14:23.374783 25299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "3cbee32b549b45e0bc9d662cf4d538ca"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c20ee65f029109323d466679e47f77f2"
server_key_iv: "8273c3006bfb7492dac63674c9ba23e8"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.375080 25299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.386224 25299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:23.388422 25299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:23.388535 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:23.388695 25299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:23.388999 25299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:23.389034 25299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.389055 25299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:23.389101 25299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.395186 25299 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:32939
I20260812 00:14:23.395263 25503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:32939 every 8 connection(s)
I20260812 00:14:23.395606 25299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:14:23.397192 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 25299
I20260812 00:14:23.397500 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:38511
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:34739
--webserver_port=46421
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:23.405471 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.396310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34543 (local address 127.31.227.62:34739)
0812 00:14:23.396373 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.396375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.396592 (+ 217us) server_negotiation.cc:407] Connection header received
0812 00:14:23.397159 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.397161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.397179 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.397231 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:23.398216 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.398705 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.399674 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.399878 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.399956 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.402530 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:23.402532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:23.402532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:23.402539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:23.403955 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.404482 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.404483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.404483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.404533 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.404823 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.404823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.404823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.404920 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:23.404971 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.405332 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.405401 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:14:23.405823 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.396282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:34543)
0812 00:14:23.396480 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.396513 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.397043 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.397297 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.397301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.397913 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:23.398116 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.398121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.398800 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.398802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.399577 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.399579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.399973 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.399974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.400090 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.400569 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:23.400577 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:23.402435 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:23.404043 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.404051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.404054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.404401 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.404601 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.404603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.404603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.404752 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.404989 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:23.404997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:23.405188 (+ 191us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.405316 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.405506 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260812 00:14:23.406419 25504 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34739
I20260812 00:14:23.406510 25504 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:23.406694 25504 heartbeater.cc:511] Master 127.31.227.62:34739 requested a full tablet report, sending...
I20260812 00:14:23.407052 24981 ts_manager.cc:194] Registered new tserver with Master: 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939)
I20260812 00:14:23.407402 24981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:34543
W20260812 00:14:23.482846 25508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.483073 25508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.483112 25508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.484259 25508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.484325 25508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.484370 25508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:14:23.485929 25508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:38511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=46421
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:23.486207 25508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.486440 25508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:23.491641 25520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.491649 25525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.491649 25522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.491943 25508 server_base.cc:1034] running on GCE node
I20260812 00:14:23.492182 25508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.492424 25508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.493566 25508 hybrid_clock.cc:648] HybridClock initialized: now 1786493663493547 us; error 29 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.496094 25508 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:14:23.496541 25508 webserver.cc:460] Webserver started at http://127.31.227.3:46421/ using document root <none> and password file <none>
I20260812 00:14:23.496779 25508 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.496862 25508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.498335 25508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:23.498966 25533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.499146 25508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:23.499238 25508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "5931c9c1a4595232eac37d7a7065979e"
server_key_iv: "6159a308696ae7475f1c1c320c45f4d6"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.499601 25508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.526854 25508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:23.529141 25508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:23.529269 25508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:23.529448 25508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:23.529780 25508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:23.529837 25508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.529881 25508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:23.529907 25508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.536134 25508 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:38511
I20260812 00:14:23.536237 25720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:38511 every 8 connection(s)
I20260812 00:14:23.536550 25508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:23.543619 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 25508
I20260812 00:14:23.545845 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.537311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:52203 (local address 127.31.227.62:34739)
0812 00:14:23.537378 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.537380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.537603 (+ 223us) server_negotiation.cc:407] Connection header received
0812 00:14:23.538186 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.538188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.538199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.538250 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:23.539210 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.539702 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.540639 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.540818 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.540869 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.542925 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:23.542926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:23.542926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:23.542933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:23.544428 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.544831 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.544833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.544833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.544898 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.545156 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.545157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.545158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.545271 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:23.545346 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.545673 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.545758 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:14:23.546001 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.537308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:52203)
0812 00:14:23.537459 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.537495 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.538084 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.538304 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.538308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.538884 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:23.539119 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.539124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.539781 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.539782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.540565 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.540567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.540895 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.540896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.540990 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.541411 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:23.541419 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:23.542835 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:23.544526 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.544533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.544535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.544751 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.544975 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:23.544976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:23.544976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:23.545080 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:23.545350 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:23.545364 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:23.545488 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.545619 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.545776 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260812 00:14:23.546627 25721 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34739
I20260812 00:14:23.546732 25721 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:23.546919 25721 heartbeater.cc:511] Master 127.31.227.62:34739 requested a full tablet report, sending...
I20260812 00:14:23.547322 24981 ts_manager.cc:194] Registered new tserver with Master: 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
I20260812 00:14:23.547851 24981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:52203
I20260812 00:14:23.550303 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.544033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.31.227.62:34739)
0812 00:14:23.544170 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.544172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.544196 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:14:23.544274 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.544275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.544286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.544366 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:23.544919 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.545390 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.546367 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.546682 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.546749 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.548128 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:23.548129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:23.548129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:23.548137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:23.549399 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.549751 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.549752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.549752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.549799 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:23.549953 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:23.549953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:23.549954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:23.550037 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:23.550154 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.550185 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.550244 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260812 00:14:23.550917 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:14:23.553293 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.551247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.31.227.62:34739)
0812 00:14:23.551297 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.551299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.551402 (+ 103us) server_negotiation.cc:407] Connection header received
0812 00:14:23.551448 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.551448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.551474 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.551529 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:23.551606 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.552034 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.552707 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.552887 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.552933 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.553068 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:23.553148 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:23.553205 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.553235 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.553252 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:14:23.553586 24981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:14:23.555191 24981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:14:23.557248 25176 tablet_service.cc:1467] Processing CreateTablet for tablet fad5da8961234abfbfce27a18a0c7b9b (DEFAULT_TABLE table=test-table [id=4be18c21192c40c5a52e610b1a5f850e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:23.557484 25176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad5da8961234abfbfce27a18a0c7b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:23.559095 25111 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap starting.
I20260812 00:14:23.559458 25111 tablet_bootstrap.cc:654] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:23.560356 25111 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: No bootstrap required, opened a new log
I20260812 00:14:23.560432 25111 ts_tablet_manager.cc:1397] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:23.560611 25111 raft_consensus.cc:348] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.560693 25111 raft_consensus.cc:374] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:23.560746 25111 raft_consensus.cc:729] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cdd2fcf958446db946e853c0fd9b48, State: Initialized, Role: FOLLOWER
I20260812 00:14:23.560793 25111 consensus_queue.cc:260] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.560894 25111 ts_tablet_manager.cc:1428] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.561156 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.557016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:32939 (local address 127.0.0.1:45880)
0812 00:14:23.557250 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.557262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.557303 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.557508 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.557509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.557511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:23.557584 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.557586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.558770 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.558770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.559832 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.559835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.560811 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.560812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.560893 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.560920 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.560954 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.561009 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":7}
I20260812 00:14:23.561298 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.557170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.31.227.3:38511)
0812 00:14:23.557563 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.557568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.557578 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:23.557611 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.557613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.557629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.557701 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:23.557880 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.558547 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.560007 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.560701 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.561100 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.561158 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.561181 (+ 23us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.561217 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":82,"threads_started":1}
I20260812 00:14:23.561471 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.557118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.31.227.2:32939)
0812 00:14:23.557324 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.557328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.557341 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:14:23.557383 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.557386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.557402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.557486 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:23.557651 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.558664 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.559925 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.560701 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.561060 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.561133 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.561162 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.561207 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260812 00:14:23.561841 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.557110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:38511 (local address 127.0.0.1:39908)
0812 00:14:23.557386 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.557394 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.557427 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.557770 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.557776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.557778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:23.557806 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.557808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.558658 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.558659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.559947 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.559949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.560811 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.560812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.560893 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.560922 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.560954 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.561008 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":8,"thread_start_us":100,"threads_started":1}
I20260812 00:14:23.562167 25402 tablet_service.cc:1467] Processing CreateTablet for tablet fad5da8961234abfbfce27a18a0c7b9b (DEFAULT_TABLE table=test-table [id=4be18c21192c40c5a52e610b1a5f850e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:23.562502 25402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad5da8961234abfbfce27a18a0c7b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:23.562532 25609 tablet_service.cc:1467] Processing CreateTablet for tablet fad5da8961234abfbfce27a18a0c7b9b (DEFAULT_TABLE table=test-table [id=4be18c21192c40c5a52e610b1a5f850e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:23.562870 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad5da8961234abfbfce27a18a0c7b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:23.565374 25742 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Bootstrap starting.
I20260812 00:14:23.565835 25744 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Bootstrap starting.
I20260812 00:14:23.566088 25742 tablet_bootstrap.cc:654] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:23.566455 25744 tablet_bootstrap.cc:654] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:23.566622 25742 log.cc:824] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:23.567009 25744 log.cc:824] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:23.567408 25742 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: No bootstrap required, opened a new log
I20260812 00:14:23.567488 25742 ts_tablet_manager.cc:1397] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:14:23.567749 25744 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: No bootstrap required, opened a new log
I20260812 00:14:23.567830 25744 ts_tablet_manager.cc:1397] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:14:23.568982 25742 raft_consensus.cc:348] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.569106 25742 raft_consensus.cc:374] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:23.569128 25742 raft_consensus.cc:729] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbee32b549b45e0bc9d662cf4d538ca, State: Initialized, Role: FOLLOWER
I20260812 00:14:23.569253 25742 consensus_queue.cc:260] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.569554 25742 ts_tablet_manager.cc:1428] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:14:23.569572 25504 heartbeater.cc:503] Master 127.31.227.62:34739 was elected leader, sending a full tablet report...
I20260812 00:14:23.569762 25744 raft_consensus.cc:348] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.569911 25744 raft_consensus.cc:374] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:23.569973 25744 raft_consensus.cc:729] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7434a2e78a634cb0a9de3c5a1a75acce, State: Initialized, Role: FOLLOWER
I20260812 00:14:23.570083 25744 consensus_queue.cc:260] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.570375 25744 ts_tablet_manager.cc:1428] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:14:23.570463 25721 heartbeater.cc:503] Master 127.31.227.62:34739 was elected leader, sending a full tablet report...
W20260812 00:14:23.646173 25505 tablet.cc:2369] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:14:23.647806 25747 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:23.647912 25747 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.648283 25747 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511), 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041)
I20260812 00:14:23.651190 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.648462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:38511 (local address 127.31.227.2:55929)
0812 00:14:23.648536 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.648548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.648592 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.648781 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.648783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.648785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:23.648869 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.648871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.649557 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.649558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.650315 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.650317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.650890 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.650891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.651001 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.651005 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.651064 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.651112 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 00:14:23.651207 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.648546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:55929 (local address 127.31.227.3:38511)
0812 00:14:23.648596 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.648599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.648657 (+ 58us) server_negotiation.cc:407] Connection header received
0812 00:14:23.648687 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.648688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.648696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.648773 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:23.648947 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.649436 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.650419 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.650806 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.650894 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.650902 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.651088 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.651133 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:14:23.651503 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.648502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:36041 (local address 127.31.227.2:46105)
0812 00:14:23.648688 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.648699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:23.648772 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:23.648999 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:23.649000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:23.649001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:23.649081 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.649082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.649787 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.649788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.650516 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:23.650518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.651204 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.651205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:23.651282 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.651287 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:14:23.651369 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 00:14:23.651405 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":116,"threads_started":1}
I20260812 00:14:23.651542 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.648569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46105 (local address 127.31.227.1:36041)
0812 00:14:23.648613 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.648615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.648748 (+ 133us) server_negotiation.cc:407] Connection header received
0812 00:14:23.648885 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.648887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.648920 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.648995 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:23.649163 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.649708 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.650612 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.651013 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.651099 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.651168 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.651412 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.651469 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:23.651800 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5da8961234abfbfce27a18a0c7b9b" candidate_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" is_pre_election: true
I20260812 00:14:23.651815 25215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5da8961234abfbfce27a18a0c7b9b" candidate_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cdd2fcf958446db946e853c0fd9b48" is_pre_election: true
I20260812 00:14:23.651958 25215 raft_consensus.cc:2454] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbee32b549b45e0bc9d662cf4d538ca in term 0.
I20260812 00:14:23.651978 25644 raft_consensus.cc:2454] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbee32b549b45e0bc9d662cf4d538ca in term 0.
I20260812 00:14:23.652163 25332 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbee32b549b45e0bc9d662cf4d538ca, 87cdd2fcf958446db946e853c0fd9b48; no voters:
I20260812 00:14:23.652299 25747 raft_consensus.cc:2781] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:14:23.652383 25747 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:14:23.652419 25747 raft_consensus.cc:3037] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:23.653163 25747 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.653312 25747 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [CANDIDATE]: Term 1 election: Requested vote from peers 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511), 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041)
I20260812 00:14:23.653465 25215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5da8961234abfbfce27a18a0c7b9b" candidate_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cdd2fcf958446db946e853c0fd9b48"
I20260812 00:14:23.653491 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5da8961234abfbfce27a18a0c7b9b" candidate_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
I20260812 00:14:23.653586 25215 raft_consensus.cc:3037] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:23.653596 25644 raft_consensus.cc:3037] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:23.654243 25215 raft_consensus.cc:2454] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbee32b549b45e0bc9d662cf4d538ca in term 1.
I20260812 00:14:23.654394 25644 raft_consensus.cc:2454] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbee32b549b45e0bc9d662cf4d538ca in term 1.
I20260812 00:14:23.654404 25332 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbee32b549b45e0bc9d662cf4d538ca, 87cdd2fcf958446db946e853c0fd9b48; no voters:
I20260812 00:14:23.654489 25747 raft_consensus.cc:2781] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:23.654632 25747 raft_consensus.cc:686] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 LEADER]: Becoming Leader. State: Replica: 3cbee32b549b45e0bc9d662cf4d538ca, State: Running, Role: LEADER
I20260812 00:14:23.654735 25747 consensus_queue.cc:237] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:23.655486 24980 catalog_manager.cc:5495] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2). New cstate: current_term: 1 leader_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } health_report { overall_health: UNKNOWN } } }
I20260812 00:14:23.678889 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.676657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.31.227.2:32939)
0812 00:14:23.676710 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:23.676712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:23.676720 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:14:23.676762 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:23.676762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:23.676779 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:23.676849 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:23.676955 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.677402 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.678189 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:23.678376 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:23.678437 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:23.678570 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:23.678658 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:23.678717 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:23.678747 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 00:14:23.678765 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:14:23.680510 25215 raft_consensus.cc:1261] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Refusing update from remote peer 3cbee32b549b45e0bc9d662cf4d538ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:23.680559 25644 raft_consensus.cc:1261] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Refusing update from remote peer 3cbee32b549b45e0bc9d662cf4d538ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:23.680819 25747 consensus_queue.cc:1035] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:23.681003 25752 consensus_queue.cc:1035] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:23.683404 25759 mvcc.cc:205] Tried to move back new op lower bound from 7317478046432952320 to 7317478046330294272. Current Snapshot: MvccSnapshot[applied={T|T < 7317478046432952320}]
I20260812 00:14:23.683646 25761 mvcc.cc:205] Tried to move back new op lower bound from 7317478046432952320 to 7317478046330294272. Current Snapshot: MvccSnapshot[applied={T|T < 7317478046432952320}]
I20260812 00:14:23.685693 24980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:23.685778 24980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:23.686467 24980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Sending DeleteTablet for 3 replicas of tablet fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:23.686761 25402 tablet_service.cc:1514] Processing DeleteTablet for tablet fad5da8961234abfbfce27a18a0c7b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:23 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:45880
I20260812 00:14:23.686805 25609 tablet_service.cc:1514] Processing DeleteTablet for tablet fad5da8961234abfbfce27a18a0c7b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:23 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:39908
I20260812 00:14:23.686983 25770 tablet_replica.cc:330] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: stopping tablet replica
I20260812 00:14:23.687038 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 25046
I20260812 00:14:23.687080 25770 raft_consensus.cc:2229] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:14:23.687176 25770 raft_consensus.cc:2258] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:23.698397 25771 tablet_replica.cc:330] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: stopping tablet replica
I20260812 00:14:23.698542 25771 raft_consensus.cc:2229] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:14:23.698658 25771 raft_consensus.cc:2258] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:23.699136 25771 ts_tablet_manager.cc:1905] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:23.699137 25770 ts_tablet_manager.cc:1905] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:23.700861 25770 ts_tablet_manager.cc:1918] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:23.700953 25770 log.cc:1196] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:23.701159 25771 ts_tablet_manager.cc:1918] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:23.701261 25771 log.cc:1196] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:23.701289 25770 ts_tablet_manager.cc:1939] T fad5da8961234abfbfce27a18a0c7b9b P 3cbee32b549b45e0bc9d662cf4d538ca: Deleting consensus metadata
I20260812 00:14:23.701678 25771 ts_tablet_manager.cc:1939] T fad5da8961234abfbfce27a18a0c7b9b P 7434a2e78a634cb0a9de3c5a1a75acce: Deleting consensus metadata
I20260812 00:14:23.701754 24955 catalog_manager.cc:4841] TS 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): tablet fad5da8961234abfbfce27a18a0c7b9b (table test-table [id=4be18c21192c40c5a52e610b1a5f850e]) successfully deleted
I20260812 00:14:23.702358 24955 catalog_manager.cc:4841] TS 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): tablet fad5da8961234abfbfce27a18a0c7b9b (table test-table [id=4be18c21192c40c5a52e610b1a5f850e]) successfully deleted
W20260812 00:14:23.704821 24958 connection.cc:535] client connection to 127.31.227.1:36041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 00:14:23.704922 24958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:14:23.705088 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 25299
I20260812 00:14:23.712904 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:23.712707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:36041 (local address 127.0.0.1:33148)
0812 00:14:23.712794 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:23.712868 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 00:14:23.713097 24958 catalog_manager.cc:4584] TS 87cdd2fcf958446db946e853c0fd9b48 (127.31.227.1:36041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fad5da8961234abfbfce27a18a0c7b9b: Network error: Client connection negotiation failed: client connection to 127.31.227.1:36041: connect: Connection refused (error 111)
I20260812 00:14:23.717430 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 25508
I20260812 00:14:23.728180 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 24934
I20260812 00:14:23.743577 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34739
--webserver_interface=127.31.227.62
--webserver_port=46075
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:23.826718 25773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.826942 25773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.826967 25773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.828166 25773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:23.828215 25773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.828230 25773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.828243 25773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:23.828255 25773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:23.829869 25773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=46075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:23.830145 25773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.830411 25773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:23.835467 25782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.835484 25783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.835496 25785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.835662 25773 server_base.cc:1034] running on GCE node
I20260812 00:14:23.835978 25773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.836294 25773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.837445 25773 hybrid_clock.cc:648] HybridClock initialized: now 1786493663837430 us; error 29 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.839958 25773 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:23.840456 25773 webserver.cc:460] Webserver started at http://127.31.227.62:46075/ using document root <none> and password file <none>
I20260812 00:14:23.840689 25773 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.840737 25773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.842201 25773 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:23.842916 25795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.843089 25773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:14:23.843199 25773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "1208c5b421e24a91be3b03f08461ab3f"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c104b11b53ded2a702782ac5490737e4"
server_key_iv: "cf3be33b22c905e8e49cb7427257e70f"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.843570 25773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.858413 25773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:23.860539 25773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:23.860708 25773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:23.865178 25773 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34739
I20260812 00:14:23.865216 25889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34739 every 8 connection(s)
I20260812 00:14:23.865717 25773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:23.867349 25892 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:14:23.868120 25892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap starting.
I20260812 00:14:23.869076 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 25773
I20260812 00:14:23.869405 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:36041
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:34739
--webserver_port=32835
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:23.878584 25892 log.cc:824] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:23.880999 25892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:14:23.881333 25892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f: Bootstrap complete.
I20260812 00:14:23.883824 25892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.884060 25892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1208c5b421e24a91be3b03f08461ab3f, State: Initialized, Role: FOLLOWER
I20260812 00:14:23.884232 25892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.884337 25892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:23.884402 25892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:23.884474 25892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:14:23.885581 25892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.885716 25892 leader_election.cc:302] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1208c5b421e24a91be3b03f08461ab3f; no voters:
I20260812 00:14:23.885934 25892 leader_election.cc:288] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 00:14:23.885974 25896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 3 FOLLOWER]: Leader election won for term 3
I20260812 00:14:23.886086 25896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [term 3 LEADER]: Becoming Leader. State: Replica: 1208c5b421e24a91be3b03f08461ab3f, State: Running, Role: LEADER
I20260812 00:14:23.886186 25896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } }
I20260812 00:14:23.886248 25892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:23.886449 25896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1208c5b421e24a91be3b03f08461ab3f. Latest consensus state: current_term: 3 leader_uuid: "1208c5b421e24a91be3b03f08461ab3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } } }
I20260812 00:14:23.886540 25896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:23.886449 25897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1208c5b421e24a91be3b03f08461ab3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1208c5b421e24a91be3b03f08461ab3f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34739 } } }
I20260812 00:14:23.886711 25897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1208c5b421e24a91be3b03f08461ab3f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:23.886752 25901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:23.887648 25901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:23.887768 25901 catalog_manager.cc:1259] Loaded cluster ID: b2a1d0815c6542d8b7619a6967246f3a
I20260812 00:14:23.887811 25901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 00:14:23.888302 25901 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9cb3fba9ad google::LogMessage::Fail() at ??:0
@ 0x7f9cb3fbd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9cb3fba4de google::LogMessage::Flush() at ??:0
@ 0x7f9cb3fbba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9cbbedf0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9cbbee88b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9cb5e880d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9cb5e7d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9cb6ce16db start_thread at ??:0
@ 0x7f9cb2ab861f clone at ??:0
W20260812 00:14:23.966486 25894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.966725 25894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.966763 25894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.968101 25894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.968163 25894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.968206 25894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:14:23.969784 25894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:36041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=32835
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:23.970055 25894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.970299 25894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:23.975798 25947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.975822 25949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.975792 25945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.976259 25894 server_base.cc:1034] running on GCE node
I20260812 00:14:23.976481 25894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.976724 25894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.977898 25894 hybrid_clock.cc:648] HybridClock initialized: now 1786493663977858 us; error 52 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.980618 25894 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:14:23.981076 25894 webserver.cc:460] Webserver started at http://127.31.227.1:32835/ using document root <none> and password file <none>
I20260812 00:14:23.981308 25894 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.981377 25894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.982957 25894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:23.983757 25963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.983960 25894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:23.984066 25894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "87cdd2fcf958446db946e853c0fd9b48"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "50a732cbcd71219f1c2cace1f0894595"
server_key_iv: "97ae7d5e4535c22b2e7b817485c49940"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.984452 25894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.999282 25894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:24.001895 25894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:24.002084 25894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:24.002370 25894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:24.002857 25971 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:14:24.004926 25894 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:14:24.005013 25894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:14:24.005061 25894 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:14:24.005882 25894 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:14:24.005942 25894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:24.006028 25971 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap starting.
I20260812 00:14:24.014062 25894 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:36041
I20260812 00:14:24.014145 26144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:36041 every 8 connection(s)
I20260812 00:14:24.014492 25894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:14:24.016225 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 25894
I20260812 00:14:24.016556 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:32939
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:34739
--webserver_port=43249
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:24.019433 25971 log.cc:824] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:24.021865 25971 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:14:24.022171 25971 tablet_bootstrap.cc:492] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Bootstrap complete.
I20260812 00:14:24.023288 25971 ts_tablet_manager.cc:1397] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260812 00:14:24.024549 25971 raft_consensus.cc:348] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:24.024817 25971 raft_consensus.cc:729] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cdd2fcf958446db946e853c0fd9b48, State: Initialized, Role: FOLLOWER
I20260812 00:14:24.024988 25971 consensus_queue.cc:260] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:24.025326 25971 ts_tablet_manager.cc:1428] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260812 00:14:24.121259 26149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:24.121454 26149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:24.121479 26149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:24.122709 26149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:24.122751 26149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:24.122777 26149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:14:24.124440 26149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:32939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=43249
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:24.124876 26149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:24.125263 26149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:14:24.130591 26149 server_base.cc:1034] running on GCE node
W20260812 00:14:24.130561 26161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:24.130558 26162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:24.130642 26165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:24.131002 26149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:24.131278 26149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:24.132462 26149 hybrid_clock.cc:648] HybridClock initialized: now 1786493664132447 us; error 32 us; skew 500 ppm
Aug 12 00:14:24 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:24.135296 26149 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:14:24.135824 26149 webserver.cc:460] Webserver started at http://127.31.227.2:43249/ using document root <none> and password file <none>
I20260812 00:14:24.136086 26149 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:24.136158 26149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:24.137610 26149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:24.138286 26182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:24.138479 26149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:24.138571 26149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "3cbee32b549b45e0bc9d662cf4d538ca"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "c20ee65f029109323d466679e47f77f2"
server_key_iv: "8273c3006bfb7492dac63674c9ba23e8"
server_key_version: "encryptionkey@0"
I20260812 00:14:24.138939 26149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:24.149348 26149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:24.151588 26149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:24.151742 26149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:24.151999 26149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:24.152346 26149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:24.152385 26149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:24.152410 26149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:24.152453 26149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:24.160308 26149 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:32939
I20260812 00:14:24.160465 26362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:32939 every 8 connection(s)
I20260812 00:14:24.160809 26149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:14:24.164039 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 26149
I20260812 00:14:24.164366 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:38511
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:34739
--webserver_port=46421
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:24.181697 26148 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:24.015241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:49959)
0812 00:14:24.015437 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.015474 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:24.015935 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:24.181446 (+165511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
W20260812 00:14:24.181979 26146 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:34739 (0 consecutive failures): Network error: Failed to ping master at 127.31.227.62:34739: Client connection negotiation failed: client connection to 127.31.227.62:34739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 00:14:24.182260 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:24.161667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:56487)
0812 00:14:24.161851 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.161889 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:24.162412 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:24.181442 (+ 19030us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260812 00:14:24.182381 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.182196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:56917)
0812 00:14:24.182270 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.182345 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 00:14:24.182644 26363 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:34739 (0 consecutive failures): Network error: Failed to ping master at 127.31.227.62:34739: Client connection negotiation failed: client connection to 127.31.227.62:34739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:14:24.182972 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.182824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:39489)
0812 00:14:24.182872 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.182939 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 00:14:24.183182 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.183075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:59371)
0812 00:14:24.183113 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.183155 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 00:14:24.183419 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.183296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:50375)
0812 00:14:24.183339 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.183384 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 00:14:24.183410 26146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:14:24.183564 26363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:14:24.253968 26370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:24.254191 26370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:24.254230 26370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:24.255375 26370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:24.255441 26370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:24.255499 26370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:14:24.256969 26370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:38511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=46421
--tserver_master_addrs=127.31.227.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:24.257246 26370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:24.257478 26370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:24.262506 26378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:24.262516 26381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:24.262516 26376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:24.262565 26370 server_base.cc:1034] running on GCE node
I20260812 00:14:24.262914 26370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:24.263180 26370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:24.264322 26370 hybrid_clock.cc:648] HybridClock initialized: now 1786493664264309 us; error 35 us; skew 500 ppm
Aug 12 00:14:24 dist-test-slave-zr1t krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:24.266845 26370 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:14:24.267271 26370 webserver.cc:460] Webserver started at http://127.31.227.3:46421/ using document root <none> and password file <none>
I20260812 00:14:24.267467 26370 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:24.267508 26370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:24.268672 26370 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:24.269191 26393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:24.269354 26370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:14:24.269465 26370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "7434a2e78a634cb0a9de3c5a1a75acce"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zr1t"
server_key: "5931c9c1a4595232eac37d7a7065979e"
server_key_iv: "6159a308696ae7475f1c1c320c45f4d6"
server_key_version: "encryptionkey@0"
I20260812 00:14:24.269840 26370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:24.282644 26370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:24.284371 26370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:24.284519 26370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:24.284729 26370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:24.285027 26370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:24.285080 26370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:24.285117 26370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:24.285145 26370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:24.291291 26370 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:38511
I20260812 00:14:24.291342 26565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:38511 every 8 connection(s)
I20260812 00:14:24.291643 26370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:14:24.292588 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.292179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:59021)
0812 00:14:24.292322 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.292425 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
W20260812 00:14:24.292954 26566 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:34739 (0 consecutive failures): Network error: Failed to ping master at 127.31.227.62:34739: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
I20260812 00:14:24.293212 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.293118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:56721)
0812 00:14:24.293165 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.293190 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 00:14:24.293430 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 26370
I20260812 00:14:24.293771 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.293679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:57125)
0812 00:14:24.293725 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:24.293748 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 00:14:24.293947 26566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:14:25.183967 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.183715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:57935)
0812 00:14:25.183867 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.183902 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260812 00:14:25.184077 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.183849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:55453)
0812 00:14:25.183965 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.183993 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260812 00:14:25.294514 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.294309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:50505)
0812 00:14:25.294427 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.294459 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260812 00:14:25.643388 26577 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:25.643512 26577 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:25.643783 26577 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939), 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
Aug 12 00:14:25 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.2@KRBTEST.COM
Aug 12 00:14:25 dist-test-slave-zr1t krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.3@KRBTEST.COM
I20260812 00:14:25.655895 26578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.644078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:38511 (local address 127.31.227.1:45949)
0812 00:14:25.644179 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.644190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.644243 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.645087 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.645088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.645278 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.645320 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.645322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.646882 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.646883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.648137 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.648140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.649051 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.649053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.649185 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.649878 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.649886 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.652290 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.654367 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.654368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.654368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.654599 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.654902 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.654902 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.654902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.654979 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.655405 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.655406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.655502 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.655607 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.655795 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":12,"thread_start_us":51,"threads_started":1}
I20260812 00:14:25.655910 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.644106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:40953 (local address 127.31.227.2:32939)
0812 00:14:25.644285 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.644291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.644302 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:14:25.644427 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.644430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.644454 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.644554 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.645317 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.646732 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.648323 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.648938 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.649057 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.652018 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.652022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.652022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.652033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.654061 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.654667 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.654670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.654671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.654771 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.655037 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.655038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.655039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.655291 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.655352 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.655676 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.655797 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260812 00:14:25.655939 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.644183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:45949 (local address 127.31.227.3:38511)
0812 00:14:25.644343 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.644354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.644386 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:14:25.644899 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.644909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.644963 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.645070 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.645422 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.646732 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.648323 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.648938 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.649057 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.652396 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.652400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.652402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.652416 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.654277 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.654699 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.654702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.654702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.654796 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.655072 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.655073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.655073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.655306 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.655407 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.655680 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.655826 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260812 00:14:25.656018 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.643970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:32939 (local address 127.31.227.1:40953)
0812 00:14:25.644068 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.644085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.644156 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.644610 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.644612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.644917 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.645185 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.645193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.646872 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.646873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.648137 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.648140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.649053 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.649054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.649185 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.649874 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.649886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.651903 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.654147 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.654155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.654158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.654583 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.654845 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.654846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.654846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.654968 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.655367 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.655375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.655503 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.655608 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.655878 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":233}
W20260812 00:14:25.657184 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
W20260812 00:14:25.657255 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:25.657306 25964 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87cdd2fcf958446db946e853c0fd9b48; no voters: 3cbee32b549b45e0bc9d662cf4d538ca, 7434a2e78a634cb0a9de3c5a1a75acce
I20260812 00:14:25.657433 26577 raft_consensus.cc:2726] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:14:26.184604 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:26.184389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:45387)
0812 00:14:26.184522 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:26.184557 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260812 00:14:26.184715 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:26.184482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:56057)
0812 00:14:26.184603 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:26.184633 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260812 00:14:26.295142 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:26.294945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:42705)
0812 00:14:26.295076 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:26.295109 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260812 00:14:27.185276 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.185012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:33467)
0812 00:14:27.185175 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:27.185212 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260812 00:14:27.185390 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.185155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:36855)
0812 00:14:27.185271 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:27.185307 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260812 00:14:27.295816 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.295627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:38527)
0812 00:14:27.295749 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:27.295781 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260812 00:14:27.415239 26590 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:27.415373 26590 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:27.415585 26590 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939), 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
W20260812 00:14:27.416057 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
W20260812 00:14:27.416139 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:27.416157 25964 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87cdd2fcf958446db946e853c0fd9b48; no voters: 3cbee32b549b45e0bc9d662cf4d538ca, 7434a2e78a634cb0a9de3c5a1a75acce
I20260812 00:14:27.416225 26590 raft_consensus.cc:2726] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:14:28.186041 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.185751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:35787)
0812 00:14:28.185931 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.185982 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260812 00:14:28.186117 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.185887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:39073)
0812 00:14:28.186021 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.186055 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260812 00:14:28.296500 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.296276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:58629)
0812 00:14:28.296408 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.296458 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260812 00:14:29.186838 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:29.186555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:46305)
0812 00:14:29.186695 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:29.186762 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260812 00:14:29.186888 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:29.186685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:42619)
0812 00:14:29.186811 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:29.186849 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260812 00:14:29.297297 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:29.297098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:52277)
0812 00:14:29.297227 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:29.297261 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260812 00:14:29.653578 26602 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:29.653673 26602 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:29.653824 26602 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939), 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
W20260812 00:14:29.654222 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
W20260812 00:14:29.654299 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:29.654315 25964 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87cdd2fcf958446db946e853c0fd9b48; no voters: 3cbee32b549b45e0bc9d662cf4d538ca, 7434a2e78a634cb0a9de3c5a1a75acce
I20260812 00:14:29.654392 26602 raft_consensus.cc:2726] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:14:30.187587 26603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.187342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:37285)
0812 00:14:30.187489 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:30.187539 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260812 00:14:30.187599 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.187371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:47149)
0812 00:14:30.187522 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:30.187557 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260812 00:14:30.297905 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.297721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:55657)
0812 00:14:30.297836 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:30.297870 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260812 00:14:31.188235 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:31.187998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:42451)
0812 00:14:31.188158 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:31.188191 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260812 00:14:31.188248 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:31.188001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:51381)
0812 00:14:31.188147 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:31.188199 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260812 00:14:31.298511 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:31.298324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:40857)
0812 00:14:31.298439 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:31.298472 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260812 00:14:32.188886 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:32.188651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:42975)
0812 00:14:32.188800 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:32.188838 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260812 00:14:32.188992 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:32.188745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:57877)
0812 00:14:32.188893 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:32.188924 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260812 00:14:32.299199 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:32.298991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:53221)
0812 00:14:32.299122 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:32.299158 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260812 00:14:33.189600 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:33.189318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:33965)
0812 00:14:33.189465 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:33.189501 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260812 00:14:33.189661 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:33.189421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:59845)
0812 00:14:33.189578 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:33.189619 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260812 00:14:33.299978 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:33.299757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:38633)
0812 00:14:33.299886 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:33.299925 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260812 00:14:33.639400 26626 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:33.639503 26626 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:33.639684 26626 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939), 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
W20260812 00:14:33.640161 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
W20260812 00:14:33.640265 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:33.640300 25964 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87cdd2fcf958446db946e853c0fd9b48; no voters: 3cbee32b549b45e0bc9d662cf4d538ca, 7434a2e78a634cb0a9de3c5a1a75acce
I20260812 00:14:33.640424 26626 raft_consensus.cc:2726] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:14:34.190294 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:34.190062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:43345)
0812 00:14:34.190203 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:34.190246 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260812 00:14:34.190295 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:34.190064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:43691)
0812 00:14:34.190219 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:34.190249 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260812 00:14:34.300760 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:34.300525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:58563)
0812 00:14:34.300668 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:34.300713 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260812 00:14:35.190971 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:35.190731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:42877)
0812 00:14:35.190862 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:35.190907 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260812 00:14:35.190973 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:35.190732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:50859)
0812 00:14:35.190906 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:35.190939 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260812 00:14:35.301656 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:35.301378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:59655)
0812 00:14:35.301579 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:35.301612 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260812 00:14:36.191608 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:36.191391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:53137)
0812 00:14:36.191531 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:36.191564 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260812 00:14:36.191658 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:36.191434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:46837)
0812 00:14:36.191580 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:36.191618 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260812 00:14:36.302426 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:36.302178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:46485)
0812 00:14:36.302351 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:36.302386 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
I20260812 00:14:37.192497 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:37.192261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:35093)
0812 00:14:37.192385 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:37.192436 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260812 00:14:37.192509 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:37.192083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:53765)
0812 00:14:37.192404 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:37.192452 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":55,"threads_started":1}
I20260812 00:14:37.303094 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:37.302865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:57009)
0812 00:14:37.303014 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:37.303051 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260812 00:14:38.193323 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:38.193119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:52249)
0812 00:14:38.193244 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:38.193281 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260812 00:14:38.193341 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:38.193142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:45383)
0812 00:14:38.193268 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:38.193301 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260812 00:14:38.303776 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:38.303569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:56651)
0812 00:14:38.303693 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:38.303729 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260812 00:14:38.670768 26651 raft_consensus.cc:482] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:38.670897 26651 raft_consensus.cc:504] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbee32b549b45e0bc9d662cf4d538ca" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 32939 } } peers { permanent_uuid: "7434a2e78a634cb0a9de3c5a1a75acce" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 38511 } } peers { permanent_uuid: "87cdd2fcf958446db946e853c0fd9b48" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 36041 } }
I20260812 00:14:38.671093 26651 leader_election.cc:288] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939), 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511)
W20260812 00:14:38.671505 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbee32b549b45e0bc9d662cf4d538ca (127.31.227.2:32939): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
W20260812 00:14:38.671607 25964 leader_election.cc:341] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7434a2e78a634cb0a9de3c5a1a75acce (127.31.227.3:38511): Not found: Tablet not found: fad5da8961234abfbfce27a18a0c7b9b
I20260812 00:14:38.671648 25964 leader_election.cc:302] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87cdd2fcf958446db946e853c0fd9b48; no voters: 3cbee32b549b45e0bc9d662cf4d538ca, 7434a2e78a634cb0a9de3c5a1a75acce
I20260812 00:14:38.671739 26651 raft_consensus.cc:2726] T fad5da8961234abfbfce27a18a0c7b9b P 87cdd2fcf958446db946e853c0fd9b48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:14:39.194142 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:39.193905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.2:37311)
0812 00:14:39.194051 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:39.194091 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260812 00:14:39.194142 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:39.193905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.1:39233)
0812 00:14:39.194051 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:39.194092 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260812 00:14:39.301797 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 25894
I20260812 00:14:39.311219 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:39.309998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34739 (local address 127.31.227.3:37067)
0812 00:14:39.311130 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:39.311168 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":970,"threads_started":1}
I20260812 00:14:39.317926 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 26149
I20260812 00:14:39.326838 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 26370
2026-08-12T00:14:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18626 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:39 dist-test-slave-zr1t krb5kdc[26661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:39 dist-test-slave-zr1t krb5kdc[26661](info): set up 2 sockets
Aug 12 00:14:39 dist-test-slave-zr1t krb5kdc[26661](info): commencing operation
krb5kdc: starting...
W20260812 00:14:40.596235 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:14:40 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:14:40.716171 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34311
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:40.798143 26678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:40.798357 26678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:40.798379 26678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:40.799540 26678 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 00:14:40.799582 26678 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 00:14:40.799606 26678 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 00:14:40.799618 26678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:40.799631 26678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:40.799643 26678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:40.799674 26678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:40.799685 26678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:40.801213 26678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:40.801469 26678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:40.801733 26678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:40.806932 26695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:40.807035 26678 server_base.cc:1034] running on GCE node
W20260812 00:14:40.806994 26692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:40.806996 26689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:40.807406 26678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:40.807736 26678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:40.808908 26678 hybrid_clock.cc:648] HybridClock initialized: now 1786493680808893 us; error 46 us; skew 500 ppm
Aug 12 00:14:40 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:40.811406 26678 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:14:40.811892 26678 webserver.cc:460] Webserver started at http://127.31.227.62:33065/ using document root <none> and password file <none>
I20260812 00:14:40.812117 26678 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:40.812186 26678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:40.812341 26678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:40.813421 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "796f8f5c71e84f55a49ac932e2f7cffe"
format_stamp: "Formatted at 2026-08-12 00:14:40 on dist-test-slave-zr1t"
server_key: "ce9104bb711082b728291fb9a5eaf6c4"
server_key_iv: "1eff97b3da628206824e1b60bd55b55b"
server_key_version: "encryptionkey@0"
I20260812 00:14:40.813864 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "796f8f5c71e84f55a49ac932e2f7cffe"
format_stamp: "Formatted at 2026-08-12 00:14:40 on dist-test-slave-zr1t"
server_key: "ce9104bb711082b728291fb9a5eaf6c4"
server_key_iv: "1eff97b3da628206824e1b60bd55b55b"
server_key_version: "encryptionkey@0"
I20260812 00:14:40.815289 26678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:40.816017 26704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:40.816205 26678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:14:40.816316 26678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "796f8f5c71e84f55a49ac932e2f7cffe"
format_stamp: "Formatted at 2026-08-12 00:14:40 on dist-test-slave-zr1t"
server_key: "ce9104bb711082b728291fb9a5eaf6c4"
server_key_iv: "1eff97b3da628206824e1b60bd55b55b"
server_key_version: "encryptionkey@0"
I20260812 00:14:40.816392 26678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:40.843426 26678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:40.845443 26678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:40.845641 26678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:40.849831 26678 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34311
I20260812 00:14:40.849867 26795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34311 every 8 connection(s)
I20260812 00:14:40.850237 26678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:40.850983 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 26678
I20260812 00:14:40.851073 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:14:40.851270 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:40.851305 32652 external_mini_cluster.cc:1428] Setting key e4bb2e915b3aa89d020335938fc0dcee
I20260812 00:14:40.854794 26796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap starting.
I20260812 00:14:40.855422 26796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:40.855719 26796 log.cc:824] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:40.856420 26796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: No bootstrap required, opened a new log
Aug 12 00:14:40 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:40.858749 26796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:14:40.858954 26796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:40.859010 26796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796f8f5c71e84f55a49ac932e2f7cffe, State: Initialized, Role: FOLLOWER
I20260812 00:14:40.859179 26796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:14:40.859267 26796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:40.859304 26796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:40.859380 26796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:40.860018 26796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:14:40.860137 26796 leader_election.cc:302] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796f8f5c71e84f55a49ac932e2f7cffe; no voters:
I20260812 00:14:40.860342 26796 leader_election.cc:288] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:40.860383 26803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:40.860548 26803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 LEADER]: Becoming Leader. State: Replica: 796f8f5c71e84f55a49ac932e2f7cffe, State: Running, Role: LEADER
I20260812 00:14:40.860612 26796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:40.860767 26803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:14:40.861168 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f8f5c71e84f55a49ac932e2f7cffe. Latest consensus state: current_term: 1 leader_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } } }
I20260812 00:14:40.861299 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:40.861836 26815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:40.862389 26815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:40.862444 26805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } } }
I20260812 00:14:40.862743 26805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:40.863273 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:40.851825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.31.227.62:34311)
0812 00:14:40.852025 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:40.852035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:40.852069 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:14:40.852552 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:40.852561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:40.852628 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:40.852784 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:40.853367 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:40.854772 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:40.855579 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:40.856099 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:40.856218 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:40.858871 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:40.858874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:40.858875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:40.858885 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:40.860869 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:40.861292 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:40.861295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:40.861295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:40.861439 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:40.861747 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:40.861749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:40.861749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:40.862262 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:40.862874 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:40.862968 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 00:14:40.863108 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260812 00:14:40.863895 26815 catalog_manager.cc:1347] Generated new cluster ID: cb3999f973e0491888a2fe435bcf4776
I20260812 00:14:40.863943 26815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:40.887365 26815 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:40.887928 26815 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:40.893098 26815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Generated new TSK 0
I20260812 00:14:40.893286 26815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:14:40.922245 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34311
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:41.005292 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:41.005496 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:41.005574 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:41.006872 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:41.006924 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:41.006950 26831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:14:41.008441 26831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:41.008695 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:41.008924 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:41.014148 26840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:41.014139 26843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:41.014140 26839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:41.014325 26831 server_base.cc:1034] running on GCE node
I20260812 00:14:41.014613 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:41.014891 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:41.016062 26831 hybrid_clock.cc:648] HybridClock initialized: now 1786493681016051 us; error 28 us; skew 500 ppm
Aug 12 00:14:41 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493681, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:41.018730 26831 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:14:41.019250 26831 webserver.cc:460] Webserver started at http://127.31.227.1:46261/ using document root <none> and password file <none>
I20260812 00:14:41.019479 26831 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:41.019547 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:41.019712 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:41.020833 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "d36c3b439584423b82a56289cfaf4cc6"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "778eda85d1ecd66db1d6630e08574a06"
server_key_iv: "4de898ad97f8b39f55ff0ed2dd1f0598"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.021207 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "d36c3b439584423b82a56289cfaf4cc6"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "778eda85d1ecd66db1d6630e08574a06"
server_key_iv: "4de898ad97f8b39f55ff0ed2dd1f0598"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.022639 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:41.023438 26856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.023628 26831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:41.023737 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "d36c3b439584423b82a56289cfaf4cc6"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "778eda85d1ecd66db1d6630e08574a06"
server_key_iv: "4de898ad97f8b39f55ff0ed2dd1f0598"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.023825 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:41.041759 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:41.044549 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:41.044737 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:41.045029 26831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:41.045379 26831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:41.045439 26831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.045486 26831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:41.045506 26831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.052906 26831 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:33267
I20260812 00:14:41.052947 27050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:33267 every 8 connection(s)
I20260812 00:14:41.053275 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:14:41.057605 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 26831
I20260812 00:14:41.057708 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:14:41.057940 32652 external_mini_cluster.cc:1428] Setting key 5da4f0affbc6fc479bfc4924227d602c
Aug 12 00:14:41 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493681, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:41.063614 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.054057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38001 (local address 127.31.227.62:34311)
0812 00:14:41.054122 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.054124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.054334 (+ 210us) server_negotiation.cc:407] Connection header received
0812 00:14:41.054834 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.054836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.054846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.054910 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:41.055897 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.056432 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.057428 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.057776 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.057858 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.060301 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:41.060303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:41.060303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:41.060312 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:41.061988 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.062593 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.062594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.062595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.062649 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.062941 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.062942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.062942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.063044 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:41.063095 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.063477 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.063548 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:14:41.063890 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.054007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:38001)
0812 00:14:41.054192 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.054236 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.054747 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.054981 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.054985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.055594 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:41.055798 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.055804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.056522 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.056523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.057319 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.057321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.057863 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.057864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.057973 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.058681 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:41.058691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:41.060199 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:41.062074 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:41.062083 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:41.062085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:41.062521 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:41.062720 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:41.062721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:41.062721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:41.062870 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:41.063118 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:41.063126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:41.063299 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.063441 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.063643 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260812 00:14:41.064441 27053 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34311
I20260812 00:14:41.064553 27053 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:41.064766 27053 heartbeater.cc:511] Master 127.31.227.62:34311 requested a full tablet report, sending...
I20260812 00:14:41.065346 26731 ts_manager.cc:194] Registered new tserver with Master: d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267)
I20260812 00:14:41.066020 26731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:38001
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:14:41.084365 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34311
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:41.168795 27062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:41.169004 27062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:41.169027 27062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:41.170176 27062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:41.170217 27062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:41.170241 27062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:14:41.171712 27062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:41.171962 27062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:41.172195 27062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:14:41.177455 27062 server_base.cc:1034] running on GCE node
W20260812 00:14:41.177407 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:41.177386 27078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:41.177379 27075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:41.177788 27062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:41.178068 27062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:41.179257 27062 hybrid_clock.cc:648] HybridClock initialized: now 1786493681179235 us; error 52 us; skew 500 ppm
Aug 12 00:14:41 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493681, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:41.181773 27062 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:14:41.182247 27062 webserver.cc:460] Webserver started at http://127.31.227.2:35243/ using document root <none> and password file <none>
I20260812 00:14:41.182479 27062 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:41.182548 27062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:41.182686 27062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:41.183724 27062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "50d5b41ca06bc6150de37aadbfbcf932"
server_key_iv: "c3ad424208695490b9129fe3be9d22cb"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.184106 27062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "50d5b41ca06bc6150de37aadbfbcf932"
server_key_iv: "c3ad424208695490b9129fe3be9d22cb"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.185482 27062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:14:41.186321 27090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.186524 27062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:41.186640 27062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "50d5b41ca06bc6150de37aadbfbcf932"
server_key_iv: "c3ad424208695490b9129fe3be9d22cb"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.186721 27062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:41.208356 27062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:41.210474 27062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:41.210645 27062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:41.210893 27062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:41.211269 27062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:41.211325 27062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.211367 27062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:41.211395 27062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.217677 27062 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:44199
I20260812 00:14:41.217726 27273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:44199 every 8 connection(s)
I20260812 00:14:41.218039 27062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:14:41.221354 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 27062
I20260812 00:14:41.221468 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:14:41.221686 32652 external_mini_cluster.cc:1428] Setting key 7aff9e368a41ec3f27c950879596d318
Aug 12 00:14:41 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493681, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:41.228896 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.218786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:58135 (local address 127.31.227.62:34311)
0812 00:14:41.218852 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.218854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.219039 (+ 185us) server_negotiation.cc:407] Connection header received
0812 00:14:41.219658 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.219659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.219669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.219729 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:41.220736 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.221246 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.222610 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.222817 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.222902 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.225257 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:41.225260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:41.225260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:41.225270 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:41.227126 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.227612 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.227613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.227614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.227681 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.228064 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.228065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.228065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.228175 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:41.228227 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.228651 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.228753 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:14:41.228930 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.218743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:58135)
0812 00:14:41.218922 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.218967 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.219564 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.219806 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.219810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.220451 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:41.220660 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.220664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.221356 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.221357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.222514 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.222517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.222921 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.222923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.223059 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.223632 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:41.223642 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:41.225157 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:41.227251 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:41.227258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:41.227260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:41.227496 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:41.227795 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:41.227797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:41.227798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:41.227974 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:41.228248 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:41.228255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:41.228450 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.228600 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.228745 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260812 00:14:41.229538 27274 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34311
I20260812 00:14:41.229656 27274 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:41.229877 27274 heartbeater.cc:511] Master 127.31.227.62:34311 requested a full tablet report, sending...
I20260812 00:14:41.230336 26731 ts_manager.cc:194] Registered new tserver with Master: fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199)
I20260812 00:14:41.230756 26731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:58135
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:14:41.251920 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34311
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:41.333947 27286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:41.334134 27286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:41.334156 27286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:41.335282 27286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:41.335342 27286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:41.335394 27286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:14:41.336797 27286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:14:41.337036 27286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:41.337316 27286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:41.342273 27292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:41.342310 27291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:41.342350 27295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:41.342415 27286 server_base.cc:1034] running on GCE node
I20260812 00:14:41.342701 27286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:41.342936 27286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:41.344077 27286 hybrid_clock.cc:648] HybridClock initialized: now 1786493681344067 us; error 27 us; skew 500 ppm
Aug 12 00:14:41 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493681, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:41.346843 27286 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:14:41.347258 27286 webserver.cc:460] Webserver started at http://127.31.227.3:45941/ using document root <none> and password file <none>
I20260812 00:14:41.347477 27286 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:41.347522 27286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:41.347676 27286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:41.348681 27286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "b9f51d73b1064577acd06707de7c59c2"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "c745fc6ce61e0ab9f915f75c5014b2e4"
server_key_iv: "7a4d4d47df1f013222f647b8ec2b5795"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.349036 27286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "b9f51d73b1064577acd06707de7c59c2"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "c745fc6ce61e0ab9f915f75c5014b2e4"
server_key_iv: "7a4d4d47df1f013222f647b8ec2b5795"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.350452 27286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:41.351191 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.351512 27286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:41.351611 27286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "b9f51d73b1064577acd06707de7c59c2"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "c745fc6ce61e0ab9f915f75c5014b2e4"
server_key_iv: "7a4d4d47df1f013222f647b8ec2b5795"
server_key_version: "encryptionkey@0"
I20260812 00:14:41.351684 27286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:41.362421 27286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:41.364545 27286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:41.364658 27286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:41.364845 27286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:14:41.365101 27286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:14:41.365137 27286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.365159 27286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:14:41.365172 27286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:41.371243 27286 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:42659
I20260812 00:14:41.371317 27512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:42659 every 8 connection(s)
I20260812 00:14:41.371649 27286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:14:41.377079 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 27286
I20260812 00:14:41.377180 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:14:41.377380 32652 external_mini_cluster.cc:1428] Setting key ed6fd646cc342093d33fdd767a3e98ce
Aug 12 00:14:41 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493681, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:14:41.381183 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.372438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:44621 (local address 127.31.227.62:34311)
0812 00:14:41.372509 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.372511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.372739 (+ 228us) server_negotiation.cc:407] Connection header received
0812 00:14:41.373237 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.373238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.373248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.373305 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:41.374281 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.374778 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.375718 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.375898 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.375949 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.378311 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:41.378312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:41.378313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:41.378320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:41.379768 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.380193 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.380195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.380195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.380258 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.380532 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.380533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.380534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.380658 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:41.380737 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.381011 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.381090 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:14:41.381385 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.372430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:44621)
0812 00:14:41.372603 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.372646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.373146 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.373358 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.373361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.373996 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:41.374195 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.374200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.374860 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.374861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.375644 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.375646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.375971 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.375972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.376078 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.376508 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:41.376517 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:41.378210 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:41.379874 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:41.379883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:41.379885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:41.380114 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:41.380332 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:41.380333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:41.380333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:41.380449 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:41.380728 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:41.380736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:41.380876 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.380989 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.381149 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260812 00:14:41.381853 27514 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34311
I20260812 00:14:41.381933 27514 heartbeater.cc:464] Registering TS with master...
I20260812 00:14:41.382076 27514 heartbeater.cc:511] Master 127.31.227.62:34311 requested a full tablet report, sending...
I20260812 00:14:41.382387 26731 ts_manager.cc:194] Registered new tserver with Master: b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
I20260812 00:14:41.382720 26731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:44621
I20260812 00:14:41.389482 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:14:41.396879 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.390859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.31.227.62:34311)
0812 00:14:41.390928 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.390930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.391012 (+ 82us) server_negotiation.cc:407] Connection header received
0812 00:14:41.391061 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.391061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.391070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.391118 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:41.391624 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.392083 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.392758 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.392938 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.392990 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.394268 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:41.394269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:41.394269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:41.394277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:41.395691 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.396058 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.396059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.396060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.396119 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:41.396328 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:41.396329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:41.396329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:41.396480 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:41.396594 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.396697 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.396778 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:14:41.398563 26731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:14:41.400038 26731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:14:41.406677 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.402259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33267 (local address 127.0.0.1:39238)
0812 00:14:41.402571 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.402613 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.402686 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.403006 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.403007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.403009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:41.403068 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.403070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.404429 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.404430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.405477 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.405478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.406459 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.406460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.406521 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.406532 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.406587 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.406625 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":63,"threads_started":1}
I20260812 00:14:41.406747 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.402371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.31.227.1:33267)
0812 00:14:41.402784 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.402788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.402797 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:41.402835 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.402837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.402853 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.402946 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:41.403161 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.404316 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.405595 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.406363 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.406439 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.406500 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.406628 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.406670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":87,"threads_started":1}
I20260812 00:14:41.407104 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.402472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:44199 (local address 127.0.0.1:39654)
0812 00:14:41.403065 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.403075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.403122 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.403347 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.403348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.403350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:41.403397 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.403399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.404618 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.404619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.405479 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.405482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.406884 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.406885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.406940 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.406943 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.406997 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.407033 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":82,"threads_started":1}
I20260812 00:14:41.407557 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.402610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.31.227.2:44199)
0812 00:14:41.402816 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.402821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.403188 (+ 367us) server_negotiation.cc:407] Connection header received
0812 00:14:41.403237 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.403239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.403255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.403331 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:41.403494 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.404516 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.405648 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.406363 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.406442 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.406500 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.407437 (+ 937us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.407484 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260812 00:14:41.407536 26953 tablet_service.cc:1467] Processing CreateTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad (DEFAULT_TABLE table=test-table [id=ac1f034c2ad249ce84db5d03d2a48d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:41.407934 26953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29011b95b2d4da19c087aa1e2f6e1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:41.408097 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.402269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:42659 (local address 127.0.0.1:51238)
0812 00:14:41.402573 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.402613 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.402688 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.402936 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.402939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.402942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:41.403019 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.403021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.404149 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.404150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.405468 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.405471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.407858 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.407859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.407922 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.407925 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.407988 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.408030 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260812 00:14:41.408145 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.402381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.31.227.3:42659)
0812 00:14:41.402677 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.402683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.402695 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:41.402812 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.402815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.402832 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.402913 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:41.403110 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.404045 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.406556 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.407692 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.407766 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.407819 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.408018 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.408064 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":82,"threads_started":1}
I20260812 00:14:41.408195 27163 tablet_service.cc:1467] Processing CreateTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad (DEFAULT_TABLE table=test-table [id=ac1f034c2ad249ce84db5d03d2a48d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:41.408540 27163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29011b95b2d4da19c087aa1e2f6e1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:41.410245 27402 tablet_service.cc:1467] Processing CreateTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad (DEFAULT_TABLE table=test-table [id=ac1f034c2ad249ce84db5d03d2a48d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:14:41.410563 27402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29011b95b2d4da19c087aa1e2f6e1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:41.411340 27541 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Bootstrap starting.
I20260812 00:14:41.411686 27542 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Bootstrap starting.
I20260812 00:14:41.412072 27541 tablet_bootstrap.cc:654] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:41.412365 27542 tablet_bootstrap.cc:654] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:41.412731 27541 log.cc:824] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:41.412923 27542 log.cc:824] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:41.413458 27547 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Bootstrap starting.
I20260812 00:14:41.413565 27541 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: No bootstrap required, opened a new log
I20260812 00:14:41.413631 27541 ts_tablet_manager.cc:1397] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:14:41.414052 27542 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: No bootstrap required, opened a new log
I20260812 00:14:41.414104 27547 tablet_bootstrap.cc:654] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:41.414145 27542 ts_tablet_manager.cc:1397] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:14:41.414588 27547 log.cc:824] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:41.415308 27547 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: No bootstrap required, opened a new log
I20260812 00:14:41.415390 27547 ts_tablet_manager.cc:1397] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 00:14:41.415707 27541 raft_consensus.cc:348] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.415907 27541 raft_consensus.cc:374] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:41.415946 27541 raft_consensus.cc:729] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36c3b439584423b82a56289cfaf4cc6, State: Initialized, Role: FOLLOWER
I20260812 00:14:41.416078 27541 consensus_queue.cc:260] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.416181 27542 raft_consensus.cc:348] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.416345 27542 raft_consensus.cc:374] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:41.416364 27541 ts_tablet_manager.cc:1428] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:14:41.416381 27542 raft_consensus.cc:729] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd333606bfa4ec6ab104d54a04a1a09, State: Initialized, Role: FOLLOWER
I20260812 00:14:41.416392 27053 heartbeater.cc:503] Master 127.31.227.62:34311 was elected leader, sending a full tablet report...
I20260812 00:14:41.416539 27542 consensus_queue.cc:260] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.416889 27274 heartbeater.cc:503] Master 127.31.227.62:34311 was elected leader, sending a full tablet report...
I20260812 00:14:41.416878 27542 ts_tablet_manager.cc:1428] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:14:41.417219 27547 raft_consensus.cc:348] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.417399 27547 raft_consensus.cc:374] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:41.417467 27547 raft_consensus.cc:729] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f51d73b1064577acd06707de7c59c2, State: Initialized, Role: FOLLOWER
I20260812 00:14:41.417606 27547 consensus_queue.cc:260] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.417907 27547 ts_tablet_manager.cc:1428] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 00:14:41.418030 27514 heartbeater.cc:503] Master 127.31.227.62:34311 was elected leader, sending a full tablet report...
I20260812 00:14:41.463032 27553 raft_consensus.cc:482] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:14:41.463155 27553 raft_consensus.cc:504] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.463551 27553 leader_election.cc:288] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267), fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199)
I20260812 00:14:41.467443 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.463934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:49719 (local address 127.31.227.2:44199)
0812 00:14:41.463980 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.463982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.464092 (+ 110us) server_negotiation.cc:407] Connection header received
0812 00:14:41.464178 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.464179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.464189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.464246 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:41.464378 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.465082 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.466506 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.467081 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.467157 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.467165 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.467335 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.467374 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:41.467443 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.463844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:42141 (local address 127.31.227.1:33267)
0812 00:14:41.463896 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.463899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.463909 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:41.463939 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.463940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.463950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.464019 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:14:41.464190 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.464992 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.466530 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.467081 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.467157 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.467166 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.467335 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.467374 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:14:41.467481 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.463718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33267 (local address 127.31.227.3:42141)
0812 00:14:41.463787 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.463805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.463868 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.464027 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.464028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.464030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:41.464109 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.464110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.465072 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.465072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.466400 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.466403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.467169 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.467170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.467258 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.467260 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.467313 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.467362 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":62}
I20260812 00:14:41.467485 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.463837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:44199 (local address 127.31.227.3:49719)
0812 00:14:41.464030 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.464040 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:41.464099 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:41.464257 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:41.464258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:41.464259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:14:41.464312 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.464313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.465153 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.465154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.466400 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:41.466403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.467169 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.467170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:41.467253 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.467257 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:14:41.467313 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:14:41.467362 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":38,"thread_start_us":129,"threads_started":1}
I20260812 00:14:41.467885 26983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29011b95b2d4da19c087aa1e2f6e1ad" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36c3b439584423b82a56289cfaf4cc6" is_pre_election: true
I20260812 00:14:41.468062 26983 raft_consensus.cc:2454] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 0.
I20260812 00:14:41.468142 27190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29011b95b2d4da19c087aa1e2f6e1ad" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" is_pre_election: true
I20260812 00:14:41.468267 27190 raft_consensus.cc:2454] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 0.
I20260812 00:14:41.468497 27316 leader_election.cc:302] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9f51d73b1064577acd06707de7c59c2, d36c3b439584423b82a56289cfaf4cc6; no voters:
W20260812 00:14:41.468603 27276 tablet.cc:2369] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:14:41.468708 27553 raft_consensus.cc:2781] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:14:41.468760 27553 raft_consensus.cc:482] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:14:41.468782 27553 raft_consensus.cc:3037] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:41.469509 27553 raft_consensus.cc:504] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.469668 27553 leader_election.cc:288] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 election: Requested vote from peers d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267), fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199)
I20260812 00:14:41.469844 27190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29011b95b2d4da19c087aa1e2f6e1ad" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
I20260812 00:14:41.469908 27190 raft_consensus.cc:3037] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:41.470629 27190 raft_consensus.cc:2454] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 1.
I20260812 00:14:41.470746 26983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29011b95b2d4da19c087aa1e2f6e1ad" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36c3b439584423b82a56289cfaf4cc6"
I20260812 00:14:41.470796 27316 leader_election.cc:302] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09; no voters:
I20260812 00:14:41.470821 26983 raft_consensus.cc:3037] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:41.470901 27553 raft_consensus.cc:2781] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:41.471081 27553 raft_consensus.cc:686] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 1 LEADER]: Becoming Leader. State: Replica: b9f51d73b1064577acd06707de7c59c2, State: Running, Role: LEADER
I20260812 00:14:41.471176 27553 consensus_queue.cc:237] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:14:41.471774 26983 raft_consensus.cc:2454] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 1.
I20260812 00:14:41.472072 26731 catalog_manager.cc:5495] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9f51d73b1064577acd06707de7c59c2 (127.31.227.3). New cstate: current_term: 1 leader_uuid: "b9f51d73b1064577acd06707de7c59c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } health_report { overall_health: HEALTHY } } }
I20260812 00:14:41.479046 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.476564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.31.227.3:42659)
0812 00:14:41.476632 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:41.476635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:41.476665 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:14:41.476758 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:41.476759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:41.476784 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:41.476854 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:41.477000 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.477421 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.478237 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:41.478432 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:41.478519 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:41.478691 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:41.478786 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:41.478881 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:41.478903 (+ 22us) server_negotiation.cc:299] Negotiation successful
0812 00:14:41.478923 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:14:41.480715 26983 raft_consensus.cc:1261] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Refusing update from remote peer b9f51d73b1064577acd06707de7c59c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:41.480715 27190 raft_consensus.cc:1261] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Refusing update from remote peer b9f51d73b1064577acd06707de7c59c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:41.481037 27553 consensus_queue.cc:1035] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:41.481174 27556 consensus_queue.cc:1035] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:41.483603 27563 mvcc.cc:205] Tried to move back new op lower bound from 7317478119342575616 to 7317478119306559488. Current Snapshot: MvccSnapshot[applied={T|T < 7317478119342575616}]
I20260812 00:14:41.483732 27564 mvcc.cc:205] Tried to move back new op lower bound from 7317478119342575616 to 7317478119306559488. Current Snapshot: MvccSnapshot[applied={T|T < 7317478119342575616}]
I20260812 00:14:41.486238 26730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:41.486333 26730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:14:41.487052 26730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Sending DeleteTablet for 3 replicas of tablet d29011b95b2d4da19c087aa1e2f6e1ad
I20260812 00:14:41.487421 26953 tablet_service.cc:1514] Processing DeleteTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:41 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:39238
I20260812 00:14:41.487466 27163 tablet_service.cc:1514] Processing DeleteTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:41 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:39654
I20260812 00:14:41.487797 27402 tablet_service.cc:1514] Processing DeleteTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:41 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:51238
I20260812 00:14:41.488013 27581 tablet_replica.cc:330] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: stopping tablet replica
I20260812 00:14:41.488026 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 26831
I20260812 00:14:41.488128 27581 raft_consensus.cc:2229] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:14:41.488312 27581 raft_consensus.cc:2258] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:41.498469 27581 ts_tablet_manager.cc:1905] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:41.498672 27580 tablet_replica.cc:330] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: stopping tablet replica
I20260812 00:14:41.498812 27580 raft_consensus.cc:2229] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:14:41.498960 27580 raft_consensus.cc:2258] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:14:41.500890 27581 ts_tablet_manager.cc:1918] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:41.501032 27581 log.cc:1196] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/d29011b95b2d4da19c087aa1e2f6e1ad
I20260812 00:14:41.501353 27581 ts_tablet_manager.cc:1939] T d29011b95b2d4da19c087aa1e2f6e1ad P b9f51d73b1064577acd06707de7c59c2: Deleting consensus metadata
I20260812 00:14:41.501492 27580 ts_tablet_manager.cc:1905] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:14:41.501952 26706 catalog_manager.cc:4841] TS b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): tablet d29011b95b2d4da19c087aa1e2f6e1ad (table test-table [id=ac1f034c2ad249ce84db5d03d2a48d9f]) successfully deleted
I20260812 00:14:41.503046 27580 ts_tablet_manager.cc:1918] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:14:41.503134 27580 log.cc:1196] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/d29011b95b2d4da19c087aa1e2f6e1ad
I20260812 00:14:41.503461 27580 ts_tablet_manager.cc:1939] T d29011b95b2d4da19c087aa1e2f6e1ad P fdd333606bfa4ec6ab104d54a04a1a09: Deleting consensus metadata
I20260812 00:14:41.503883 26711 catalog_manager.cc:4841] TS fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): tablet d29011b95b2d4da19c087aa1e2f6e1ad (table test-table [id=ac1f034c2ad249ce84db5d03d2a48d9f]) successfully deleted
W20260812 00:14:41.506784 26711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:33267 (error 108)
I20260812 00:14:41.507139 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 27062
I20260812 00:14:41.509054 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:41.507233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33267 (local address 127.0.0.1:39254)
0812 00:14:41.508959 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:41.509009 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:33267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1704}
W20260812 00:14:41.510303 26711 catalog_manager.cc:4584] TS d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d29011b95b2d4da19c087aa1e2f6e1ad: Network error: Client connection negotiation failed: client connection to 127.31.227.1:33267: connect: Connection refused (error 111)
I20260812 00:14:41.519691 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 27286
I20260812 00:14:41.537158 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 26678
I20260812 00:15:01.552994 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34311
--webserver_interface=127.31.227.62
--webserver_port=33065
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:01.637639 27587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:01.637840 27587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:01.637862 27587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:01.639017 27587 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 00:15:01.639058 27587 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 00:15:01.639072 27587 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 00:15:01.639084 27587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:01.639097 27587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:01.639151 27587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:01.639169 27587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:01.639189 27587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:01.640682 27587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:01.640937 27587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:01.641199 27587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:15:01.651937 27587 server_base.cc:1034] running on GCE node
W20260812 00:15:01.651989 27596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:01.651964 27595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:01.651953 27598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:01.652513 27587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:01.652802 27587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:01.653941 27587 hybrid_clock.cc:648] HybridClock initialized: now 1786493701653930 us; error 34 us; skew 500 ppm
Aug 12 00:15:01 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:01.656523 27587 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:15:01.657033 27587 webserver.cc:460] Webserver started at http://127.31.227.62:33065/ using document root <none> and password file <none>
I20260812 00:15:01.657280 27587 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:01.657363 27587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:01.658883 27587 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:01.659693 27608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:01.659889 27587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:01.659997 27587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "796f8f5c71e84f55a49ac932e2f7cffe"
format_stamp: "Formatted at 2026-08-12 00:14:40 on dist-test-slave-zr1t"
server_key: "ce9104bb711082b728291fb9a5eaf6c4"
server_key_iv: "1eff97b3da628206824e1b60bd55b55b"
server_key_version: "encryptionkey@0"
I20260812 00:15:01.660423 27587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:01.674093 27587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:01.675920 27587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:01.676091 27587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:01.680572 27587 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34311
I20260812 00:15:01.680619 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34311 every 8 connection(s)
I20260812 00:15:01.681041 27587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:01.682570 27701 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:15:01.683192 27701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap starting.
I20260812 00:15:01.689429 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 27587
I20260812 00:15:01.689720 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:33267
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:34311
--webserver_port=46261
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:15:01.693081 27701 log.cc:824] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:01.695021 27701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:15:01.695331 27701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap complete.
I20260812 00:15:01.697711 27701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:01.697965 27701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796f8f5c71e84f55a49ac932e2f7cffe, State: Initialized, Role: FOLLOWER
I20260812 00:15:01.698146 27701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:01.698222 27701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:01.698292 27701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:01.698355 27701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:15:01.699529 27701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:01.699656 27701 leader_election.cc:302] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796f8f5c71e84f55a49ac932e2f7cffe; no voters:
I20260812 00:15:01.699852 27701 leader_election.cc:288] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:15:01.699890 27708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:15:01.700012 27708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 LEADER]: Becoming Leader. State: Replica: 796f8f5c71e84f55a49ac932e2f7cffe, State: Running, Role: LEADER
I20260812 00:15:01.700150 27701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:01.700155 27708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:01.700609 27709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f8f5c71e84f55a49ac932e2f7cffe. Latest consensus state: current_term: 2 leader_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } } }
I20260812 00:15:01.700701 27709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:01.700922 27711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } } }
I20260812 00:15:01.701045 27711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:01.702157 27735 catalog_manager.cc:1259] Loaded cluster ID: cb3999f973e0491888a2fe435bcf4776
I20260812 00:15:01.702207 27735 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: loading cluster ID for follower catalog manager: success
I20260812 00:15:01.703369 27735 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: acquiring CA information for follower catalog manager: success
I20260812 00:15:01.703786 27735 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:15:01.703866 27719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:01.704229 27719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:01.704288 27719 catalog_manager.cc:1259] Loaded cluster ID: cb3999f973e0491888a2fe435bcf4776
I20260812 00:15:01.704342 27719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:01.704491 27719 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:01.704576 27719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Loaded TSK: 0
I20260812 00:15:01.727243 27719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Generated new TSK 1
I20260812 00:15:01.727355 27719 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:15:01.777205 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:01.777439 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:01.777479 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:01.778637 27704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:01.778705 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:01.778750 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:01.780241 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:33267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=46261
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:01.780505 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:01.780766 27704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:01.786952 27746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:01.787070 27745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:01.787007 27748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:01.787007 27704 server_base.cc:1034] running on GCE node
I20260812 00:15:01.787427 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:01.787738 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:01.788901 27704 hybrid_clock.cc:648] HybridClock initialized: now 1786493701788868 us; error 58 us; skew 500 ppm
Aug 12 00:15:01 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:01.791667 27704 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:15:01.792191 27704 webserver.cc:460] Webserver started at http://127.31.227.1:46261/ using document root <none> and password file <none>
I20260812 00:15:01.792457 27704 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:01.792537 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:01.794461 27704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:15:01.795306 27759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:01.795563 27704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:01.795663 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "d36c3b439584423b82a56289cfaf4cc6"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "778eda85d1ecd66db1d6630e08574a06"
server_key_iv: "4de898ad97f8b39f55ff0ed2dd1f0598"
server_key_version: "encryptionkey@0"
I20260812 00:15:01.796144 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:01.808818 27704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:01.811275 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:01.811429 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:01.811704 27704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:01.812229 27767 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:15:01.813870 27704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:15:01.813942 27704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:15:01.813979 27704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:15:01.814836 27704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:15:01.814894 27704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 00:15:01.814997 27767 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Bootstrap starting.
I20260812 00:15:01.822667 27704 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:33267
I20260812 00:15:01.822741 27939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:33267 every 8 connection(s)
I20260812 00:15:01.823208 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:15:01.826617 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 27704
I20260812 00:15:01.826922 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:44199
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:34311
--webserver_port=35243
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:15:01.827261 27767 log.cc:824] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:01.829829 27767 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:15:01.830152 27767 tablet_bootstrap.cc:492] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Bootstrap complete.
I20260812 00:15:01.831997 27767 ts_tablet_manager.cc:1397] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
Aug 12 00:15:01 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:01.833444 27767 raft_consensus.cc:348] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:15:01.833796 27767 raft_consensus.cc:729] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36c3b439584423b82a56289cfaf4cc6, State: Initialized, Role: FOLLOWER
I20260812 00:15:01.834275 27767 consensus_queue.cc:260] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } } peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } }
I20260812 00:15:01.834605 27767 ts_tablet_manager.cc:1428] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:01.837579 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:01.823961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:38789)
0812 00:15:01.824206 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:01.824255 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:01.824886 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:01.826155 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:01.826160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:01.826508 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:01.826778 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:01.826786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.828537 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.828538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:01.829832 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:01.829836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.830692 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.830694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:01.830843 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:01.831544 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:01.831554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:01.833431 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:01.835747 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:01.835754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:01.835756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:01.835992 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:01.836304 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:01.836305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:01.836305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:01.836410 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:01.837007 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:01.837013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:01.837135 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 00:15:01.837220 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:15:01.837372 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":136,"threads_started":1}
I20260812 00:15:01.837778 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:01.824109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38789 (local address 127.31.227.62:34311)
0812 00:15:01.825663 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:01.825675 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:01.825726 (+ 51us) server_negotiation.cc:407] Connection header received
0812 00:15:01.825872 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:01.825880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:01.825941 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:01.826107 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:01.826955 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.828382 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.829962 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.830558 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.831264 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:01.833609 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:01.833614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:01.833615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:01.833628 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:01.835668 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:01.836084 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:01.836088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:01.836088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:01.836229 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:01.836478 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:01.836479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:01.836479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:01.836809 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:01.836902 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:01.837318 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 00:15:01.837492 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":80,"threads_started":1}
I20260812 00:15:01.838584 27944 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34311
I20260812 00:15:01.838686 27944 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:01.838868 27944 heartbeater.cc:511] Master 127.31.227.62:34311 requested a full tablet report, sending...
I20260812 00:15:01.839726 27642 ts_manager.cc:194] Registered new tserver with Master: d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267)
I20260812 00:15:01.841096 27642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:38789
I20260812 00:15:01.842120 27944 heartbeater.cc:503] Master 127.31.227.62:34311 was elected leader, sending a full tablet report...
I20260812 00:15:01.850492 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:01.840706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.31.227.1:33267)
0812 00:15:01.845917 (+ 5211us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:01.845923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:01.845934 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:15:01.845971 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:01.845973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:01.845988 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:01.846078 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:01.846671 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.847772 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.849257 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.850036 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.850111 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:01.850164 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:01.850365 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 00:15:01.850405 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5174,"thread_start_us":83,"threads_started":1}
I20260812 00:15:01.850528 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:01.840441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33267 (local address 127.0.0.1:41700)
0812 00:15:01.845679 (+ 5238us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:01.845722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:01.845812 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:01.846102 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:01.846106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:01.846108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:01.846587 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:01.846590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.847871 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.847873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:01.849160 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:01.849164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.850128 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.850129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:01.850236 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:01.850250 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:15:01.850342 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 00:15:01.850422 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5194,"thread_start_us":98,"threads_started":1}
I20260812 00:15:01.851176 27821 tablet_service.cc:1514] Processing DeleteTablet for tablet d29011b95b2d4da19c087aa1e2f6e1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:14:41 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:41700
I20260812 00:15:01.851843 27958 tablet_replica.cc:330] stopping tablet replica
I20260812 00:15:01.851953 27958 raft_consensus.cc:2229] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:01.852023 27958 raft_consensus.cc:2258] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:01.852437 27958 ts_tablet_manager.cc:1905] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:01.854678 27958 ts_tablet_manager.cc:1918] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:01.854758 27958 log.cc:1196] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal/wals/d29011b95b2d4da19c087aa1e2f6e1ad
I20260812 00:15:01.855049 27958 ts_tablet_manager.cc:1939] T d29011b95b2d4da19c087aa1e2f6e1ad P d36c3b439584423b82a56289cfaf4cc6: Deleting consensus metadata
I20260812 00:15:01.855613 27617 catalog_manager.cc:4841] TS d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267): tablet d29011b95b2d4da19c087aa1e2f6e1ad (table test-table [id=ac1f034c2ad249ce84db5d03d2a48d9f]) successfully deleted
W20260812 00:15:01.920338 27951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:01.920533 27951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:01.920557 27951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:01.921645 27951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:01.921679 27951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:01.921705 27951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:01.923187 27951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:44199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=35243
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:01.923383 27951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:01.923558 27951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:01.928692 27968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:01.928718 27971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:01.928704 27969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:01.928984 27951 server_base.cc:1034] running on GCE node
I20260812 00:15:01.929177 27951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:01.929419 27951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:01.930603 27951 hybrid_clock.cc:648] HybridClock initialized: now 1786493701930579 us; error 37 us; skew 500 ppm
Aug 12 00:15:01 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:01.933207 27951 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:15:01.933672 27951 webserver.cc:460] Webserver started at http://127.31.227.2:35243/ using document root <none> and password file <none>
I20260812 00:15:01.933871 27951 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:01.933926 27951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:01.935210 27951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:01.935770 27984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:01.935945 27951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:01.936054 27951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "50d5b41ca06bc6150de37aadbfbcf932"
server_key_iv: "c3ad424208695490b9129fe3be9d22cb"
server_key_version: "encryptionkey@0"
I20260812 00:15:01.936398 27951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:01.942904 27951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:01.945133 27951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:01.945307 27951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:01.945567 27951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:01.945895 27951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:01.945951 27951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:01.945992 27951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:01.946018 27951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:01.952219 27951 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:44199
I20260812 00:15:01.952299 28168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:44199 every 8 connection(s)
I20260812 00:15:01.952629 27951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:15:01.957597 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 27951
I20260812 00:15:01.957885 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:42659
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:34311
--webserver_port=45941
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:15:01 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:01.962527 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:01.953326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60699 (local address 127.31.227.62:34311)
0812 00:15:01.953392 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:01.953395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:01.953652 (+ 257us) server_negotiation.cc:407] Connection header received
0812 00:15:01.954161 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:01.954162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:01.954177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:01.954234 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:01.955511 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.956133 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.957095 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.957287 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.957337 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:01.959581 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:01.959583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:01.959583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:01.959591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:01.960947 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:01.961445 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:01.961446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:01.961446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:01.961499 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:01.961858 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:01.961858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:01.961859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:01.961962 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:01.962014 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:01.962339 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 00:15:01.962460 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:15:01.962781 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:01.953312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:60699)
0812 00:15:01.953502 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:01.953576 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:01.954051 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:01.954286 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:01.954289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:01.955090 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:01.955414 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:01.955423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.956222 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.956223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:01.957020 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:01.957022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:01.957393 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:01.957394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:01.957500 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:01.958100 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:01.958108 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:01.959474 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:01.961027 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:01.961034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:01.961036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:01.961362 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:01.961637 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:01.961638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:01.961639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:01.961783 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:01.962027 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:01.962034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:01.962196 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 00:15:01.962327 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 00:15:01.962526 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
I20260812 00:15:01.963308 28169 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34311
I20260812 00:15:01.963486 28169 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:01.963671 28169 heartbeater.cc:511] Master 127.31.227.62:34311 requested a full tablet report, sending...
I20260812 00:15:01.964154 27642 ts_manager.cc:194] Registered new tserver with Master: fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199)
I20260812 00:15:01.964527 27642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:60699
W20260812 00:15:02.042531 28183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:02.042848 28183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:02.042889 28183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:02.043983 28183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:02.044018 28183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:02.044041 28183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:02.045466 28183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:42659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=45941
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:02.045723 28183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:02.045917 28183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:02.050992 28191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:02.051067 28192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:02.051045 28197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:02.051261 28183 server_base.cc:1034] running on GCE node
I20260812 00:15:02.051476 28183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:02.051714 28183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:02.052887 28183 hybrid_clock.cc:648] HybridClock initialized: now 1786493702052874 us; error 27 us; skew 500 ppm
Aug 12 00:15:02 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:02.055608 28183 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:15:02.056044 28183 webserver.cc:460] Webserver started at http://127.31.227.3:45941/ using document root <none> and password file <none>
I20260812 00:15:02.056293 28183 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:02.056388 28183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:02.057850 28183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:02.058496 28213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:02.058676 28183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:15:02.058787 28183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "b9f51d73b1064577acd06707de7c59c2"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "c745fc6ce61e0ab9f915f75c5014b2e4"
server_key_iv: "7a4d4d47df1f013222f647b8ec2b5795"
server_key_version: "encryptionkey@0"
I20260812 00:15:02.059146 28183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:02.069082 28183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:02.070910 28183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:02.071079 28183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:02.071317 28183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:02.071636 28183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:02.071691 28183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:02.071723 28183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:02.071750 28183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:02.078017 28183 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:42659
I20260812 00:15:02.078073 28407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:42659 every 8 connection(s)
I20260812 00:15:02.078415 28183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:15:02 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:02.084996 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 28183
I20260812 00:15:02.087666 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.079110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:56183 (local address 127.31.227.62:34311)
0812 00:15:02.079158 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.079160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.079371 (+ 211us) server_negotiation.cc:407] Connection header received
0812 00:15:02.079880 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.079881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.079892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.079949 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:02.080821 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.081285 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.082304 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.082496 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.082555 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.084512 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:02.084514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:02.084514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:02.084521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:02.086204 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:02.086652 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:02.086654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:02.086654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:02.086725 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:02.086989 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:02.086990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:02.086991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:02.087115 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:02.087188 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.087479 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.087573 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:15:02.087841 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.079073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:56183)
0812 00:15:02.079246 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:02.079287 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:02.079792 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:02.080000 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:02.080003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:02.080558 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:02.080742 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.080747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.081382 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.081382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.082209 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.082211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.082574 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.082575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.082674 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.083092 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:02.083101 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:02.084426 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:02.086321 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:02.086333 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:02.086335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:02.086567 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:02.086797 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:02.086798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:02.086798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:02.086911 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:02.087192 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:02.087200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:02.087337 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 00:15:02.087442 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:15:02.087603 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260812 00:15:02.088323 28409 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34311
I20260812 00:15:02.088444 28409 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:02.088649 28409 heartbeater.cc:511] Master 127.31.227.62:34311 requested a full tablet report, sending...
I20260812 00:15:02.089129 27642 ts_manager.cc:194] Registered new tserver with Master: b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
I20260812 00:15:02.089712 27642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:56183
I20260812 00:15:02.092157 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.085409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.31.227.62:34311)
0812 00:15:02.085607 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.085609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.085725 (+ 116us) server_negotiation.cc:407] Connection header received
0812 00:15:02.085818 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.085818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.085832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.085926 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:02.086527 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.087016 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.087827 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.088166 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.088235 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.089725 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:02.089727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:02.089727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:02.089734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:02.091317 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:02.091616 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:02.091617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:02.091617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:02.091669 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:02.091809 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:02.091810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:02.091810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:02.091899 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:02.092008 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.092033 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.092094 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":43,"threads_started":1}
I20260812 00:15:02.092934 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:02.095330 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.093286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.31.227.62:34311)
0812 00:15:02.093320 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.093322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.093421 (+ 99us) server_negotiation.cc:407] Connection header received
0812 00:15:02.093481 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.093482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.093508 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.093606 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:02.093698 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.094129 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.094804 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.094985 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.095033 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.095157 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:02.095194 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 00:15:02.095291 (+ 97us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42238: token expired
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 00:15:02.095399 28414 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42238: token expired
W20260812 00:15:02.095575 27518 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.227.62:34311)
I20260812 00:15:02.101214 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.095840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.31.227.62:34311)
0812 00:15:02.095872 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.095874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.095914 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:15:02.095930 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.095931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.095937 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.095972 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:02.096372 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.096782 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.097422 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.097645 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.097689 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.098661 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:02.098662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:02.098663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:02.098668 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:02.100014 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:02.100387 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:02.100388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:02.100389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:02.100443 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:02.100676 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:02.100677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:02.100677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:02.100801 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:02.100895 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.101028 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.101089 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 00:15:02.102352 27642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:02.103820 27642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:02.105937 27821 tablet_service.cc:1467] Processing CreateTablet for tablet e93b7adb317c4758a2c0ac2cb3beef08 (DEFAULT_TABLE table=test-table [id=70424b34f2f545559c86ff25704a7ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:02.106146 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93b7adb317c4758a2c0ac2cb3beef08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:02.107685 27767 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Bootstrap starting.
I20260812 00:15:02.108042 27767 tablet_bootstrap.cc:654] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:02.108639 27767 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: No bootstrap required, opened a new log
I20260812 00:15:02.108696 27767 ts_tablet_manager.cc:1397] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:02.108814 27767 raft_consensus.cc:348] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.108865 27767 raft_consensus.cc:374] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:02.108901 27767 raft_consensus.cc:729] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36c3b439584423b82a56289cfaf4cc6, State: Initialized, Role: FOLLOWER
I20260812 00:15:02.108959 27767 consensus_queue.cc:260] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.109050 27767 ts_tablet_manager.cc:1428] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:02.109763 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.105910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.31.227.3:42659)
0812 00:15:02.106125 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.106130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.106139 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:02.106177 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.106180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.106195 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.106270 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:02.106384 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.107191 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.108762 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.109401 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.109482 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.109574 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.109669 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.109709 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260812 00:15:02.109764 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.105633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:42659 (local address 127.0.0.1:47394)
0812 00:15:02.105803 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:02.105812 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:02.105852 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:02.106269 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:02.106270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:02.106272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:02.106327 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.106329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.107283 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.107284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.108651 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.108659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.109493 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.109494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.109585 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.109589 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:02.109655 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:15:02.109693 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260812 00:15:02.110505 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.105709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.31.227.2:44199)
0812 00:15:02.105973 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.105978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.105986 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:15:02.106042 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.106045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.106063 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.106153 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:02.106312 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.107351 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.109178 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.110195 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.110253 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.110288 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.110423 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.110449 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20260812 00:15:02.110530 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.105615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:44199 (local address 127.0.0.1:52814)
0812 00:15:02.105734 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:02.105744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:02.105777 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:02.106169 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:02.106170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:02.106172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:02.106212 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.106213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.107436 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.107436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.108450 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.108453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.110273 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.110275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.110339 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.110343 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:02.110414 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:15:02.110456 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 00:15:02.110494 28295 tablet_service.cc:1467] Processing CreateTablet for tablet e93b7adb317c4758a2c0ac2cb3beef08 (DEFAULT_TABLE table=test-table [id=70424b34f2f545559c86ff25704a7ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:02.110870 28295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93b7adb317c4758a2c0ac2cb3beef08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:02.111162 28067 tablet_service.cc:1467] Processing CreateTablet for tablet e93b7adb317c4758a2c0ac2cb3beef08 (DEFAULT_TABLE table=test-table [id=70424b34f2f545559c86ff25704a7ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:02.111478 28067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93b7adb317c4758a2c0ac2cb3beef08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:02.113741 28426 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Bootstrap starting.
I20260812 00:15:02.114168 28427 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Bootstrap starting.
I20260812 00:15:02.114434 28426 tablet_bootstrap.cc:654] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:02.114862 28427 tablet_bootstrap.cc:654] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:02.114992 28426 log.cc:824] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:02.115520 28427 log.cc:824] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:02.115900 28426 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: No bootstrap required, opened a new log
I20260812 00:15:02.115993 28426 ts_tablet_manager.cc:1397] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:02.116237 28427 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: No bootstrap required, opened a new log
I20260812 00:15:02.116291 28427 ts_tablet_manager.cc:1397] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:02.117789 28426 raft_consensus.cc:348] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.117959 28426 raft_consensus.cc:374] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:02.118006 28426 raft_consensus.cc:729] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f51d73b1064577acd06707de7c59c2, State: Initialized, Role: FOLLOWER
I20260812 00:15:02.118041 28427 raft_consensus.cc:348] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.118127 28426 consensus_queue.cc:260] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.118204 28427 raft_consensus.cc:374] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:02.118229 28427 raft_consensus.cc:729] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd333606bfa4ec6ab104d54a04a1a09, State: Initialized, Role: FOLLOWER
I20260812 00:15:02.118359 28427 consensus_queue.cc:260] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.118455 28426 ts_tablet_manager.cc:1428] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 00:15:02.118580 28409 heartbeater.cc:503] Master 127.31.227.62:34311 was elected leader, sending a full tablet report...
I20260812 00:15:02.118660 28427 ts_tablet_manager.cc:1428] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:15:02.118664 28169 heartbeater.cc:503] Master 127.31.227.62:34311 was elected leader, sending a full tablet report...
W20260812 00:15:02.203261 28171 tablet.cc:2369] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:02.326323 27945 tablet.cc:2369] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:02.328951 28410 tablet.cc:2369] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:15:02.359071 28431 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:02.359205 28431 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.359534 28431 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267)
I20260812 00:15:02.362778 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.359806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:44199 (local address 127.31.227.3:34785)
0812 00:15:02.359889 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:02.359904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:02.359992 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:02.360186 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:02.360188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:02.360190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:02.360273 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.360276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.361001 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.361002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.361873 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.361875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.362535 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.362536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.362623 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.362628 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:02.362683 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:15:02.362721 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 00:15:02.362840 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.359904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:34785 (local address 127.31.227.2:44199)
0812 00:15:02.359973 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.359975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.359989 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:15:02.360061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.360062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.360073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.360164 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:02.360364 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.360917 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.361960 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.362458 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.362519 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.362527 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.362699 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.362741 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:15:02.362943 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.359938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33267 (local address 127.31.227.3:44351)
0812 00:15:02.360123 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:02.360132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:02.360182 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:02.360362 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:02.360363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:02.360365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:02.360443 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.360444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.361093 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.361094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.361988 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:02.361991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.362655 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.362674 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:02.362749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.362754 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:02.362815 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:15:02.362852 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260812 00:15:02.362982 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.360013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:44351 (local address 127.31.227.1:33267)
0812 00:15:02.360136 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.360138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.360172 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:15:02.360246 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.360246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.360258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.360334 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:02.360519 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.361010 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.362078 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.362571 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.362647 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.362655 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.362838 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.362891 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260812 00:15:02.363185 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93b7adb317c4758a2c0ac2cb3beef08" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" is_pre_election: true
I20260812 00:15:02.363343 28097 raft_consensus.cc:2454] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 0.
I20260812 00:15:02.363510 27878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93b7adb317c4758a2c0ac2cb3beef08" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36c3b439584423b82a56289cfaf4cc6" is_pre_election: true
I20260812 00:15:02.363596 28216 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09; no voters:
I20260812 00:15:02.363665 27878 raft_consensus.cc:2454] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 0.
I20260812 00:15:02.363751 28431 raft_consensus.cc:2781] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:02.363816 28431 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:02.363847 28431 raft_consensus.cc:3037] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:02.364641 28431 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.364774 28431 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 election: Requested vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267)
I20260812 00:15:02.364970 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93b7adb317c4758a2c0ac2cb3beef08" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
I20260812 00:15:02.364979 27878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93b7adb317c4758a2c0ac2cb3beef08" candidate_uuid: "b9f51d73b1064577acd06707de7c59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36c3b439584423b82a56289cfaf4cc6"
I20260812 00:15:02.365056 28097 raft_consensus.cc:3037] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:02.365056 27878 raft_consensus.cc:3037] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:02.365981 28097 raft_consensus.cc:2454] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 1.
I20260812 00:15:02.365981 27878 raft_consensus.cc:2454] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f51d73b1064577acd06707de7c59c2 in term 1.
I20260812 00:15:02.366175 28216 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09; no voters:
I20260812 00:15:02.366273 28431 raft_consensus.cc:2781] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:02.366417 28431 raft_consensus.cc:686] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 1 LEADER]: Becoming Leader. State: Replica: b9f51d73b1064577acd06707de7c59c2, State: Running, Role: LEADER
I20260812 00:15:02.366539 28431 consensus_queue.cc:237] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:02.367333 27642 catalog_manager.cc:5495] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9f51d73b1064577acd06707de7c59c2 (127.31.227.3). New cstate: current_term: 1 leader_uuid: "b9f51d73b1064577acd06707de7c59c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } health_report { overall_health: UNKNOWN } } }
I20260812 00:15:02.391387 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.388847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.31.227.3:42659)
0812 00:15:02.388926 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:02.388928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:02.388937 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:02.388969 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:02.388970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:02.388987 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:02.389049 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:02.389168 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.389659 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.390568 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:02.390767 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:02.390858 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:02.391048 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:02.391143 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:02.391235 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:02.391255 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 00:15:02.391275 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:15:02.393224 28097 raft_consensus.cc:1261] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Refusing update from remote peer b9f51d73b1064577acd06707de7c59c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:02.393258 27878 raft_consensus.cc:1261] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Refusing update from remote peer b9f51d73b1064577acd06707de7c59c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:02.393668 28439 consensus_queue.cc:1035] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:02.393808 28431 consensus_queue.cc:1035] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:02.398248 27642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:02.398358 27642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:02.399075 27642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Sending DeleteTablet for 3 replicas of tablet e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:02.399387 27821 tablet_service.cc:1514] Processing DeleteTablet for tablet e93b7adb317c4758a2c0ac2cb3beef08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:02 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:41700
I20260812 00:15:02.399376 28067 tablet_service.cc:1514] Processing DeleteTablet for tablet e93b7adb317c4758a2c0ac2cb3beef08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:02 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:52814
I20260812 00:15:02.399592 28295 tablet_service.cc:1514] Processing DeleteTablet for tablet e93b7adb317c4758a2c0ac2cb3beef08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:02 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:47394
I20260812 00:15:02.399622 28462 tablet_replica.cc:330] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: stopping tablet replica
I20260812 00:15:02.399682 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 27704
I20260812 00:15:02.399724 28462 raft_consensus.cc:2229] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:02.399819 28462 raft_consensus.cc:2258] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:02.400108 28464 tablet_replica.cc:330] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: stopping tablet replica
I20260812 00:15:02.400242 28464 raft_consensus.cc:2229] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:15:02.400413 28464 raft_consensus.cc:2258] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:02.402797 28462 ts_tablet_manager.cc:1905] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:02.405018 28462 ts_tablet_manager.cc:1918] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:02.405112 28462 log.cc:1196] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:02.405485 28462 ts_tablet_manager.cc:1939] T e93b7adb317c4758a2c0ac2cb3beef08 P fdd333606bfa4ec6ab104d54a04a1a09: Deleting consensus metadata
I20260812 00:15:02.405917 28464 ts_tablet_manager.cc:1905] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:02.405925 27617 catalog_manager.cc:4841] TS fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): tablet e93b7adb317c4758a2c0ac2cb3beef08 (table test-table [id=70424b34f2f545559c86ff25704a7ee0]) successfully deleted
I20260812 00:15:02.407644 28464 ts_tablet_manager.cc:1918] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:02.407778 28464 log.cc:1196] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:02.408152 28464 ts_tablet_manager.cc:1939] T e93b7adb317c4758a2c0ac2cb3beef08 P b9f51d73b1064577acd06707de7c59c2: Deleting consensus metadata
I20260812 00:15:02.408653 27609 catalog_manager.cc:4841] TS b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): tablet e93b7adb317c4758a2c0ac2cb3beef08 (table test-table [id=70424b34f2f545559c86ff25704a7ee0]) successfully deleted
W20260812 00:15:02.414278 27617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:33267 (error 108)
I20260812 00:15:02.414649 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 27951
I20260812 00:15:02.415275 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:02.414758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:33267 (local address 127.0.0.1:41706)
0812 00:15:02.415191 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:02.415236 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:33267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412}
W20260812 00:15:02.429481 27617 catalog_manager.cc:4584] TS d36c3b439584423b82a56289cfaf4cc6 (127.31.227.1:33267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e93b7adb317c4758a2c0ac2cb3beef08: Network error: Client connection negotiation failed: client connection to 127.31.227.1:33267: connect: Connection refused (error 111)
I20260812 00:15:02.435580 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 28183
I20260812 00:15:02.444682 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 27587
I20260812 00:15:22.458395 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34311
--webserver_interface=127.31.227.62
--webserver_port=33065
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:22.542348 28470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:22.542559 28470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:22.542583 28470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:22.543644 28470 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 00:15:22.543677 28470 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 00:15:22.543690 28470 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 00:15:22.543702 28470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:22.543715 28470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:22.543727 28470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:22.543740 28470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:22.543751 28470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:22.545261 28470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:22.545454 28470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:22.545634 28470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:22.550765 28482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.550760 28485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.550758 28481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:22.550787 28470 server_base.cc:1034] running on GCE node
I20260812 00:15:22.551229 28470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:22.551524 28470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:22.552667 28470 hybrid_clock.cc:648] HybridClock initialized: now 1786493722552659 us; error 32 us; skew 500 ppm
Aug 12 00:15:22 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:22.558008 28470 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:15:22.558508 28470 webserver.cc:460] Webserver started at http://127.31.227.62:33065/ using document root <none> and password file <none>
I20260812 00:15:22.558732 28470 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:22.558797 28470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:22.560175 28470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:22.560890 28499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:22.561059 28470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:22.561172 28470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "796f8f5c71e84f55a49ac932e2f7cffe"
format_stamp: "Formatted at 2026-08-12 00:14:40 on dist-test-slave-zr1t"
server_key: "ce9104bb711082b728291fb9a5eaf6c4"
server_key_iv: "1eff97b3da628206824e1b60bd55b55b"
server_key_version: "encryptionkey@0"
I20260812 00:15:22.561573 28470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:22.574380 28470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:22.576038 28470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:22.576207 28470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:22.580292 28470 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34311
I20260812 00:15:22.580348 28584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34311 every 8 connection(s)
I20260812 00:15:22.580726 28470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:22.582131 28585 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:15:22.582741 28585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap starting.
I20260812 00:15:22.584463 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 28470
I20260812 00:15:22.584736 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:33267
--local_ip_for_outbound_sockets=127.31.227.1
--tserver_master_addrs=127.31.227.62:34311
--webserver_port=46261
--webserver_interface=127.31.227.1
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:15:22.593842 28585 log.cc:824] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:22.597553 28585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:15:22.597908 28585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Bootstrap complete.
I20260812 00:15:22.600246 28585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:22.600507 28585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796f8f5c71e84f55a49ac932e2f7cffe, State: Initialized, Role: FOLLOWER
I20260812 00:15:22.600677 28585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:22.600771 28585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:22.600819 28585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:22.600867 28585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:15:22.602210 28585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:22.602355 28585 leader_election.cc:302] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796f8f5c71e84f55a49ac932e2f7cffe; no voters:
I20260812 00:15:22.602559 28585 leader_election.cc:288] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 00:15:22.602602 28591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 3 FOLLOWER]: Leader election won for term 3
I20260812 00:15:22.602735 28591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [term 3 LEADER]: Becoming Leader. State: Replica: 796f8f5c71e84f55a49ac932e2f7cffe, State: Running, Role: LEADER
I20260812 00:15:22.602831 28585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:22.602849 28591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } }
I20260812 00:15:22.603117 28592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } } }
I20260812 00:15:22.603116 28591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f8f5c71e84f55a49ac932e2f7cffe. Latest consensus state: current_term: 3 leader_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f8f5c71e84f55a49ac932e2f7cffe" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34311 } } }
I20260812 00:15:22.603223 28592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:22.603240 28591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:22.603554 28601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:22.604365 28601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:22.604477 28601 catalog_manager.cc:1259] Loaded cluster ID: cb3999f973e0491888a2fe435bcf4776
I20260812 00:15:22.604518 28601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:22.605644 28601 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:22.605832 28601 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Loaded TSK: 0
I20260812 00:15:22.605885 28601 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 796f8f5c71e84f55a49ac932e2f7cffe: Loaded TSK: 1
F20260812 00:15:22.606127 28601 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2326cbe9ad google::LogMessage::Fail() at ??:0
@ 0x7f2326cc188a google::LogMessage::SendToLog() at ??:0
@ 0x7f2326cbe4de google::LogMessage::Flush() at ??:0
@ 0x7f2326cbfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f232942b104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2329428d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f232ec1c1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f232ec1c48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f232ec1c594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f232ebe2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f232ebecad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2328b8c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2328b81954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f23299e56db start_thread at ??:0
@ 0x7f23257bc61f clone at ??:0
W20260812 00:15:22.700480 28589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:22.700766 28589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:22.700809 28589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:22.702592 28589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:22.702654 28589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:22.702700 28589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:22.704962 28589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:33267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=46261
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:22.705228 28589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:22.705533 28589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:22.711028 28651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.711153 28652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.711011 28655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:22.711310 28589 server_base.cc:1034] running on GCE node
I20260812 00:15:22.711648 28589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:22.711946 28589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:22.713112 28589 hybrid_clock.cc:648] HybridClock initialized: now 1786493722713087 us; error 42 us; skew 500 ppm
Aug 12 00:15:22 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:22.715948 28589 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:15:22.716388 28589 webserver.cc:460] Webserver started at http://127.31.227.1:46261/ using document root <none> and password file <none>
I20260812 00:15:22.716658 28589 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:22.716732 28589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:22.718307 28589 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:15:22.719071 28667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:22.719246 28589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:22.719357 28589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "d36c3b439584423b82a56289cfaf4cc6"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "778eda85d1ecd66db1d6630e08574a06"
server_key_iv: "4de898ad97f8b39f55ff0ed2dd1f0598"
server_key_version: "encryptionkey@0"
I20260812 00:15:22.719729 28589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:22.739617 28589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:22.742141 28589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:22.742332 28589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:22.742594 28589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:22.743067 28677 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:15:22.745146 28589 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:15:22.745214 28589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:15:22.745273 28589 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:15:22.746011 28589 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:15:22.746068 28589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 00:15:22.746145 28677 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Bootstrap starting.
I20260812 00:15:22.753650 28589 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:33267
I20260812 00:15:22.753786 28851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:33267 every 8 connection(s)
I20260812 00:15:22.754088 28589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:15:22.758536 28677 log.cc:824] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:22.761435 28677 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:15:22.761826 28677 tablet_bootstrap.cc:492] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Bootstrap complete.
I20260812 00:15:22.762035 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 28589
I20260812 00:15:22.762264 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:44199
--local_ip_for_outbound_sockets=127.31.227.2
--tserver_master_addrs=127.31.227.62:34311
--webserver_port=35243
--webserver_interface=127.31.227.2
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:15:22.763303 28677 ts_tablet_manager.cc:1397] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260812 00:15:22.764802 28677 raft_consensus.cc:348] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:22.765158 28677 raft_consensus.cc:729] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36c3b439584423b82a56289cfaf4cc6, State: Initialized, Role: FOLLOWER
I20260812 00:15:22.765338 28677 consensus_queue.cc:260] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:22.765671 28677 ts_tablet_manager.cc:1428] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260812 00:15:22.827735 28855 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:15:22.754799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:34411)
0812 00:15:22.754974 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:22.755018 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:22.755466 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:22.827468 (+ 72002us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
W20260812 00:15:22.828052 28853 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:34311 (0 consecutive failures): Network error: Failed to ping master at 127.31.227.62:34311: Client connection negotiation failed: client connection to 127.31.227.62:34311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:15:22.828464 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:22.828319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:39379)
0812 00:15:22.828381 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:22.828432 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:15:22.828851 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:22.828746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:42013)
0812 00:15:22.828796 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:22.828828 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 00:15:22.829010 28853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:15:22.847396 28858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:22.847590 28858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:22.847613 28858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:22.848750 28858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:22.848793 28858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:22.848819 28858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:22.850379 28858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:44199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=35243
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:22.850617 28858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:22.850847 28858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:15:22.855957 28858 server_base.cc:1034] running on GCE node
W20260812 00:15:22.855870 28868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.855882 28871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.855871 28874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:22.856375 28858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:22.856647 28858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:22.857813 28858 hybrid_clock.cc:648] HybridClock initialized: now 1786493722857805 us; error 35 us; skew 500 ppm
Aug 12 00:15:22 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:22.860234 28858 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:15:22.860718 28858 webserver.cc:460] Webserver started at http://127.31.227.2:35243/ using document root <none> and password file <none>
I20260812 00:15:22.860945 28858 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:22.861016 28858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:22.862381 28858 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:22.862980 28887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:22.863154 28858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:22.863260 28858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "fdd333606bfa4ec6ab104d54a04a1a09"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "50d5b41ca06bc6150de37aadbfbcf932"
server_key_iv: "c3ad424208695490b9129fe3be9d22cb"
server_key_version: "encryptionkey@0"
I20260812 00:15:22.863605 28858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:22.873353 28858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:22.875172 28858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:22.875336 28858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:22.875589 28858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:22.875897 28858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:22.875953 28858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:22.875986 28858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:22.876031 28858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:22.882443 28858 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:44199
I20260812 00:15:22.882493 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:44199 every 8 connection(s)
I20260812 00:15:22.882829 28858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:15:22.883893 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:22.883489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:46017)
0812 00:15:22.883683 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:22.883755 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
W20260812 00:15:22.884442 29064 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:34311 (0 consecutive failures): Network error: Failed to ping master at 127.31.227.62:34311: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
I20260812 00:15:22.884712 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:22.884618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:37907)
0812 00:15:22.884664 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:22.884692 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 00:15:22.884958 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:22.884908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:38961)
0812 00:15:22.884929 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:22.884947 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260812 00:15:22.885068 29064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:15:22.888497 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 28858
I20260812 00:15:22.888814 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:42659
--local_ip_for_outbound_sockets=127.31.227.3
--tserver_master_addrs=127.31.227.62:34311
--webserver_port=45941
--webserver_interface=127.31.227.3
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:22.973460 29070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:22.973742 29070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:22.973768 29070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:22.974946 29070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:22.974988 29070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:22.975015 29070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:22.976509 29070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:42659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=45941
--tserver_master_addrs=127.31.227.62:34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:22.976756 29070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:22.976991 29070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:22.982074 29082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:22.982082 29078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:22.982401 29070 server_base.cc:1034] running on GCE node
W20260812 00:15:22.982101 29089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:22.982679 29070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:22.982934 29070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:22.984097 29070 hybrid_clock.cc:648] HybridClock initialized: now 1786493722984084 us; error 29 us; skew 500 ppm
Aug 12 00:15:22 dist-test-slave-zr1t krb5kdc[26661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:22.986670 29070 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:15:22.987150 29070 webserver.cc:460] Webserver started at http://127.31.227.3:45941/ using document root <none> and password file <none>
I20260812 00:15:22.987399 29070 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:22.987473 29070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:22.988821 29070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:22.989394 29098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:22.989670 29070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:22.989782 29070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "b9f51d73b1064577acd06707de7c59c2"
format_stamp: "Formatted at 2026-08-12 00:14:41 on dist-test-slave-zr1t"
server_key: "c745fc6ce61e0ab9f915f75c5014b2e4"
server_key_iv: "7a4d4d47df1f013222f647b8ec2b5795"
server_key_version: "encryptionkey@0"
I20260812 00:15:22.990149 29070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:23.025864 29070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:23.027894 29070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:23.028070 29070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:23.028333 29070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:23.028671 29070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:23.028728 29070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:23.028769 29070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:23.028815 29070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:23.035075 29070 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:42659
I20260812 00:15:23.035122 29289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:42659 every 8 connection(s)
I20260812 00:15:23.035532 29070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:15:23.036744 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:23.036343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:53369)
0812 00:15:23.036526 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:23.036600 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
W20260812 00:15:23.037197 29291 heartbeater.cc:644] Failed to heartbeat to 127.31.227.62:34311 (0 consecutive failures): Network error: Failed to ping master at 127.31.227.62:34311: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
I20260812 00:15:23.037595 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:23.037395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:46549)
0812 00:15:23.037471 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:23.037509 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 00:15:23.037956 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:23.037877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:34883)
0812 00:15:23.037915 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:23.037940 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 00:15:23.038120 29291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:15:23.045750 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 29070
I20260812 00:15:23.829694 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:23.829395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:60289)
0812 00:15:23.829590 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:23.829631 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
I20260812 00:15:23.885715 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:23.885414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:60179)
0812 00:15:23.885591 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:23.885638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260812 00:15:24.038643 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.038440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:33539)
0812 00:15:24.038561 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:24.038594 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260812 00:15:24.125698 29299 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:24.125849 29299 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:24.126205 29299 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
Aug 12 00:15:24 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.3@KRBTEST.COM
Aug 12 00:15:24 dist-test-slave-zr1t krb5kdc[26661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.2@KRBTEST.COM
I20260812 00:15:24.136571 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.126578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38971 (local address 127.31.227.3:42659)
0812 00:15:24.126731 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:24.126740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:24.126769 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:15:24.127287 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:24.127289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:24.127342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:24.127514 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:24.128262 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.129117 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.130216 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.130636 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.130759 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:24.133018 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:24.133023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:24.133023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:24.133033 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:24.134920 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135358 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:24.135361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:24.135362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:24.135450 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135705 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:24.135706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:24.135706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:24.135942 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:24.136032 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:24.136319 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 00:15:24.136454 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260812 00:15:24.136571 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.126381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:42659 (local address 127.31.227.1:38971)
0812 00:15:24.126444 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:24.126459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:24.126535 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:24.127535 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:24.127538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:24.127916 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:24.128112 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:24.128119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.129233 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.129234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:24.130119 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:24.130122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.130730 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.130731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:24.130850 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:24.131442 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:24.131450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:24.132902 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:24.135024 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:24.135034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:24.135254 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:24.135526 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:24.135527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:24.135633 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:24.136011 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:24.136018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:24.136151 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 00:15:24.136252 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 00:15:24.136440 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":53}
I20260812 00:15:24.136606 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.126560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:51555 (local address 127.31.227.2:44199)
0812 00:15:24.126726 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:24.126739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:24.126773 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:15:24.127286 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:24.127289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:24.127334 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:24.127483 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:24.128245 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.129108 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.130220 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.130635 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.130759 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:24.133388 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:24.133392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:24.133397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:24.133407 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:24.135143 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135527 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:24.135530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:24.135531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:24.135621 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135827 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:24.135828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:24.135828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:24.136060 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:24.136150 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:24.136339 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 00:15:24.136483 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":50,"threads_started":1}
I20260812 00:15:24.136668 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.126395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:44199 (local address 127.31.227.1:51555)
0812 00:15:24.126545 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:24.126554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:24.126600 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:24.127534 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:24.127537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:24.127917 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:24.128112 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:24.128119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.129219 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.129220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:24.130120 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:24.130123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:24.130734 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:24.130735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:24.130846 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:24.131501 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:24.131505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:24.133271 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:24.135258 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:24.135259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:24.135443 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:24.135690 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:24.135691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:24.135691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:24.135757 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:24.136134 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:24.136135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:24.136184 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:15:24.136271 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 00:15:24.136519 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":186,"thread_start_us":62,"threads_started":1}
W20260812 00:15:24.137709 28671 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
W20260812 00:15:24.137820 28668 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:24.137869 28668 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d36c3b439584423b82a56289cfaf4cc6; no voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09
I20260812 00:15:24.137967 29299 raft_consensus.cc:2726] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:15:24.830473 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.830228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:50407)
0812 00:15:24.830390 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:24.830425 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260812 00:15:24.886330 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:24.886133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:52169)
0812 00:15:24.886254 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:24.886291 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260812 00:15:25.039440 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:25.039151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:58599)
0812 00:15:25.039331 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:25.039385 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
I20260812 00:15:25.831483 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:25.831244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:56619)
0812 00:15:25.831395 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:25.831435 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260812 00:15:25.886951 29311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:25.886735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:44563)
0812 00:15:25.886874 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:25.886913 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260812 00:15:25.962585 29312 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:25.962708 29312 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:25.962914 29312 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
W20260812 00:15:25.963327 28668 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
W20260812 00:15:25.963421 28671 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:25.963502 28671 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d36c3b439584423b82a56289cfaf4cc6; no voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09
I20260812 00:15:25.963601 29312 raft_consensus.cc:2726] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:15:26.040110 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:26.039875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:52853)
0812 00:15:26.040031 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:26.040069 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260812 00:15:26.832275 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:26.832034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:54113)
0812 00:15:26.832180 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:26.832228 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260812 00:15:26.887738 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:26.887484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:40921)
0812 00:15:26.887643 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:26.887697 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260812 00:15:27.040988 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:27.040720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:51997)
0812 00:15:27.040880 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:27.040939 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260812 00:15:27.833324 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:27.833064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:34523)
0812 00:15:27.833195 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:27.833239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260812 00:15:27.888401 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:27.888196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:55665)
0812 00:15:27.888316 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:27.888358 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260812 00:15:28.041772 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:28.041505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:45851)
0812 00:15:28.041690 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:28.041730 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":128,"threads_started":1}
I20260812 00:15:28.833945 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:28.833741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:51143)
0812 00:15:28.833865 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:28.833897 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260812 00:15:28.889051 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:28.888854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:33547)
0812 00:15:28.888970 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:28.889006 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260812 00:15:29.042372 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:29.042153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:43845)
0812 00:15:29.042279 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:29.042335 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260812 00:15:29.315367 29328 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:29.315470 29328 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:29.315667 29328 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
W20260812 00:15:29.316059 28671 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
W20260812 00:15:29.316143 28668 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:29.316205 28668 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d36c3b439584423b82a56289cfaf4cc6; no voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09
I20260812 00:15:29.316285 29328 raft_consensus.cc:2726] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:15:29.834589 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:29.834353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:53051)
0812 00:15:29.834510 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:29.834541 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260812 00:15:29.889844 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:29.889602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:42327)
0812 00:15:29.889753 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:29.889790 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260812 00:15:30.043080 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:30.042842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:52947)
0812 00:15:30.042989 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:30.043021 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260812 00:15:30.835269 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:30.835028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:54289)
0812 00:15:30.835184 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:30.835223 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260812 00:15:30.890515 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:30.890323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:33081)
0812 00:15:30.890443 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:30.890476 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260812 00:15:31.043938 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:31.043636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:39879)
0812 00:15:31.043807 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:31.043867 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260812 00:15:31.836066 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:31.835807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:58917)
0812 00:15:31.835981 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:31.836018 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260812 00:15:31.891214 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:31.890947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:55507)
0812 00:15:31.891090 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:31.891132 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260812 00:15:32.044607 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:32.044359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:53227)
0812 00:15:32.044526 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:32.044563 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260812 00:15:32.263563 29347 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:32.263662 29347 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:32.263867 29347 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
W20260812 00:15:32.264410 28668 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
W20260812 00:15:32.264484 28671 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:32.264534 28671 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d36c3b439584423b82a56289cfaf4cc6; no voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09
I20260812 00:15:32.264611 29347 raft_consensus.cc:2726] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:15:32.836905 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:32.836548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:32839)
0812 00:15:32.836816 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:32.836855 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":79,"threads_started":1}
I20260812 00:15:32.891821 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:32.891619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:53997)
0812 00:15:32.891738 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:32.891776 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260812 00:15:33.045400 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:33.045146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:46183)
0812 00:15:33.045276 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:33.045322 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260812 00:15:33.837812 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:33.837576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:51255)
0812 00:15:33.837729 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:33.837765 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260812 00:15:33.892592 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:33.892296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:58721)
0812 00:15:33.892510 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:33.892548 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":146,"threads_started":1}
I20260812 00:15:34.046038 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:34.045821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:55867)
0812 00:15:34.045963 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:34.045997 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260812 00:15:34.838483 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:34.838258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:42203)
0812 00:15:34.838399 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:34.838435 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260812 00:15:34.893200 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:34.892988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:58709)
0812 00:15:34.893129 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:34.893160 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260812 00:15:35.046733 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:35.046472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:53667)
0812 00:15:35.046628 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:35.046670 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260812 00:15:35.839159 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:35.838949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:40029)
0812 00:15:35.839074 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:35.839110 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260812 00:15:35.893940 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:35.893679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:43335)
0812 00:15:35.893827 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:35.893873 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260812 00:15:36.047461 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:36.047219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:48911)
0812 00:15:36.047378 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:36.047415 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260812 00:15:36.839850 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:36.839608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:45295)
0812 00:15:36.839767 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:36.839804 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260812 00:15:36.894615 29369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:36.894394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:42417)
0812 00:15:36.894519 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:36.894572 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260812 00:15:37.048169 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:37.047896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:45597)
0812 00:15:37.048070 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:37.048115 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260812 00:15:37.086776 29372 raft_consensus.cc:482] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:37.086898 29372 raft_consensus.cc:504] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd333606bfa4ec6ab104d54a04a1a09" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 44199 } } peers { permanent_uuid: "b9f51d73b1064577acd06707de7c59c2" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 42659 } } peers { permanent_uuid: "d36c3b439584423b82a56289cfaf4cc6" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 33267 } }
I20260812 00:15:37.087106 29372 leader_election.cc:288] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199), b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659)
W20260812 00:15:37.087483 28671 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdd333606bfa4ec6ab104d54a04a1a09 (127.31.227.2:44199): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
W20260812 00:15:37.087574 28668 leader_election.cc:341] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9f51d73b1064577acd06707de7c59c2 (127.31.227.3:42659): Not found: Tablet not found: e93b7adb317c4758a2c0ac2cb3beef08
I20260812 00:15:37.087680 28668 leader_election.cc:302] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d36c3b439584423b82a56289cfaf4cc6; no voters: b9f51d73b1064577acd06707de7c59c2, fdd333606bfa4ec6ab104d54a04a1a09
I20260812 00:15:37.087769 29372 raft_consensus.cc:2726] T e93b7adb317c4758a2c0ac2cb3beef08 P d36c3b439584423b82a56289cfaf4cc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:15:37.840601 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:37.840380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.1:33107)
0812 00:15:37.840521 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:37.840554 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260812 00:15:37.895255 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:37.895057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.2:35257)
0812 00:15:37.895177 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:37.895210 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260812 00:15:38.048970 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:38.048749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34311 (local address 127.31.227.3:40287)
0812 00:15:38.048876 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:38.048929 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.62:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260812 00:15:38.049182 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 28589
I20260812 00:15:38.068364 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 28858
I20260812 00:15:38.080068 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 29070
2026-08-12T00:15:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58758 ms)
[----------] 30 tests from SecurityITest (241360 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:38 dist-test-slave-zr1t krb5kdc[29385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:38 dist-test-slave-zr1t krb5kdc[29385](info): set up 2 sockets
Aug 12 00:15:38 dist-test-slave-zr1t krb5kdc[29385](info): commencing operation
krb5kdc: starting...
W20260812 00:15:39.354033 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:39 dist-test-slave-zr1t krb5kdc[29385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:15:39.478181 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46789
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:46789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:39.560804 29401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:39.560997 29401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:39.561018 29401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:39.562214 29401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:39.562258 29401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:39.562273 29401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:39.562351 29401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:39.562378 29401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:39.563870 29401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:46789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:39.564126 29401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:39.564356 29401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:39.569506 29412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:39.569514 29414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:39.569584 29418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:39.569608 29401 server_base.cc:1034] running on GCE node
I20260812 00:15:39.570000 29401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:39.570271 29401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:39.571406 29401 hybrid_clock.cc:648] HybridClock initialized: now 1786493739571387 us; error 36 us; skew 500 ppm
Aug 12 00:15:39 dist-test-slave-zr1t krb5kdc[29385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:39.573967 29401 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:15:39.574374 29401 webserver.cc:460] Webserver started at http://127.31.227.62:37815/ using document root <none> and password file <none>
I20260812 00:15:39.574590 29401 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:39.574652 29401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:39.574779 29401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:39.575762 29401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "fffc0184d7b44f38917a615e50090f00"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "e157630e6f8c38c347c9e05440c2812d"
server_key_iv: "9c3c5fd6565e50255ac29762c28ab3b2"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.576109 29401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "fffc0184d7b44f38917a615e50090f00"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "e157630e6f8c38c347c9e05440c2812d"
server_key_iv: "9c3c5fd6565e50255ac29762c28ab3b2"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.577355 29401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:39.578120 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.578293 29401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.578388 29401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "fffc0184d7b44f38917a615e50090f00"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "e157630e6f8c38c347c9e05440c2812d"
server_key_iv: "9c3c5fd6565e50255ac29762c28ab3b2"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.578476 29401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:39.602061 29401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:39.604137 29401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:39.604331 29401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:39.608537 29401 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:46789
I20260812 00:15:39.608577 29508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:46789 every 8 connection(s)
I20260812 00:15:39.608976 29401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:39.609988 29509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:39.612353 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: Bootstrap starting.
I20260812 00:15:39.612982 29509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:39.613296 29509 log.cc:824] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:39.613479 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 29401
I20260812 00:15:39.613618 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:39.613818 32652 external_mini_cluster.cc:1428] Setting key cb7d492445a612e96de3ca7e6ae8ab07
I20260812 00:15:39.614229 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: No bootstrap required, opened a new log
I20260812 00:15:39.616410 29509 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc0184d7b44f38917a615e50090f00" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46789 } }
I20260812 00:15:39.616595 29509 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:39.616636 29509 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc0184d7b44f38917a615e50090f00, State: Initialized, Role: FOLLOWER
I20260812 00:15:39.616792 29509 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc0184d7b44f38917a615e50090f00" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46789 } }
I20260812 00:15:39.616891 29509 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:39.616923 29509 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:39.616971 29509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:39.617690 29509 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc0184d7b44f38917a615e50090f00" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46789 } }
I20260812 00:15:39.617799 29509 leader_election.cc:302] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffc0184d7b44f38917a615e50090f00; no voters:
I20260812 00:15:39.617956 29509 leader_election.cc:288] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:39.618018 29519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:39.618189 29509 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:39.618228 29519 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [term 1 LEADER]: Becoming Leader. State: Replica: fffc0184d7b44f38917a615e50090f00, State: Running, Role: LEADER
I20260812 00:15:39.619172 29519 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc0184d7b44f38917a615e50090f00" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46789 } }
I20260812 00:15:39.619858 29520 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffc0184d7b44f38917a615e50090f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc0184d7b44f38917a615e50090f00" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46789 } } }
I20260812 00:15:39.619978 29520 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:39.620605 29522 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffc0184d7b44f38917a615e50090f00. Latest consensus state: current_term: 1 leader_uuid: "fffc0184d7b44f38917a615e50090f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc0184d7b44f38917a615e50090f00" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 46789 } } }
I20260812 00:15:39.620677 29522 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00 [sys.catalog]: This master's current role is: LEADER
W20260812 00:15:39.621012 29541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:39.621079 29541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:39.621177 29546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:39.621388 29546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:15:39 dist-test-slave-zr1t krb5kdc[29385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:39.623579 29546 catalog_manager.cc:1347] Generated new cluster ID: d0158eea07fa450d84a47d52463201ec
I20260812 00:15:39.623669 29546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:39.626740 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:39.614341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.31.227.62:46789)
0812 00:15:39.614546 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:39.614555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:39.614590 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:15:39.615588 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:39.615599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:39.615665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:39.615819 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:39.616420 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:39.618186 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:39.619080 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:39.619720 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:39.619837 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:39.622813 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:39.622817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:39.622818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:39.622830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:39.625008 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:39.625437 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:39.625440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:39.625440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:39.625565 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:39.625770 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:39.625770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:39.625771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:39.626062 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:39.626215 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:39.626389 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 00:15:39.626517 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260812 00:15:39.633190 29546 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:39.633771 29546 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:39.647032 29546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: Generated new TSK 0
I20260812 00:15:39.647267 29546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:15:39.677083 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46789
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:39.761323 29553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:39.761550 29553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:39.761600 29553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:39.762763 29553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:39.762805 29553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:39.762854 29553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:39.764286 29553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:39.764515 29553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:39.764755 29553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:39.769698 29565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:39.769687 29566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:39.769680 29574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:39.769778 29553 server_base.cc:1034] running on GCE node
I20260812 00:15:39.770068 29553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:39.770329 29553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:39.771505 29553 hybrid_clock.cc:648] HybridClock initialized: now 1786493739771505 us; error 36 us; skew 500 ppm
Aug 12 00:15:39 dist-test-slave-zr1t krb5kdc[29385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:39.774255 29553 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:15:39.774745 29553 webserver.cc:460] Webserver started at http://127.31.227.1:39743/ using document root <none> and password file <none>
I20260812 00:15:39.774977 29553 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:39.775043 29553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:39.775198 29553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:39.776165 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "57aeaba97934431fa401ddb34158b386"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "73dc03b81740aecaaaacc60fa04182f2"
server_key_iv: "76fa2f8c453f78f0f1f0fbfbfff06225"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.776522 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "57aeaba97934431fa401ddb34158b386"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "73dc03b81740aecaaaacc60fa04182f2"
server_key_iv: "76fa2f8c453f78f0f1f0fbfbfff06225"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.777909 29553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:39.778649 29585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.778946 29553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:15:39.779057 29553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "57aeaba97934431fa401ddb34158b386"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "73dc03b81740aecaaaacc60fa04182f2"
server_key_iv: "76fa2f8c453f78f0f1f0fbfbfff06225"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.779139 29553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:39.796551 29553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:39.798338 29553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:39.798499 29553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:39.798720 29553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:39.799014 29553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:39.799069 29553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.799108 29553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:39.799134 29553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.805239 29553 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:41883
I20260812 00:15:39.805296 29756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:41883 every 8 connection(s)
I20260812 00:15:39.805588 29553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:15:39 dist-test-slave-zr1t krb5kdc[29385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:39.813082 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 29553
I20260812 00:15:39.813190 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:15:39.813452 32652 external_mini_cluster.cc:1428] Setting key 59f629923d6a84e08086ec258a6ba8d8
I20260812 00:15:39.814563 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:39.806270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:41025 (local address 127.31.227.62:46789)
0812 00:15:39.806330 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:39.806332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:39.806541 (+ 209us) server_negotiation.cc:407] Connection header received
0812 00:15:39.807045 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:39.807047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:39.807060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:39.807121 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:39.808014 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:39.808504 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:39.809443 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:39.809728 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:39.809804 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:39.811774 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:39.811776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:39.811776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:39.811783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:39.812900 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:39.813387 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:39.813388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:39.813388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:39.813441 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:39.813800 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:39.813801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:39.813801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:39.813967 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:39.814034 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:39.814332 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 00:15:39.814430 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:15:39.814572 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:39.806252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46789 (local address 127.31.227.1:41025)
0812 00:15:39.806427 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:39.806459 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:39.806935 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:39.807187 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:39.807191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:39.807722 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:39.807895 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:39.807899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:39.808603 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:39.808604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:39.809351 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:39.809353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:39.809814 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:39.809815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:39.809934 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:39.810373 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:39.810381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:39.811690 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:39.812982 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:39.812990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:39.812992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:39.813311 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:39.813543 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:39.813544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:39.813544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:39.813708 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:39.814035 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:39.814042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:39.814216 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 00:15:39.814300 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:15:39.814420 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260812 00:15:39.815150 29758 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46789
I20260812 00:15:39.815255 29758 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:39.815418 29758 heartbeater.cc:511] Master 127.31.227.62:46789 requested a full tablet report, sending...
I20260812 00:15:39.815972 29459 ts_manager.cc:194] Registered new tserver with Master: 57aeaba97934431fa401ddb34158b386 (127.31.227.1:41883)
I20260812 00:15:39.816674 29459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:41025
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:15:39.838140 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46789
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:39.921391 29766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:39.921650 29766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:39.921693 29766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:39.922886 29766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:39.922956 29766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:39.923002 29766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:39.924578 29766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:39.924867 29766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:39.925107 29766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:39.930251 29775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:39.930229 29777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:39.930233 29783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:39.930656 29766 server_base.cc:1034] running on GCE node
I20260812 00:15:39.930882 29766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:39.931138 29766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:39.932305 29766 hybrid_clock.cc:648] HybridClock initialized: now 1786493739932294 us; error 29 us; skew 500 ppm
Aug 12 00:15:39 dist-test-slave-zr1t krb5kdc[29385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:39.934903 29766 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:15:39.935324 29766 webserver.cc:460] Webserver started at http://127.31.227.2:35839/ using document root <none> and password file <none>
I20260812 00:15:39.935580 29766 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:39.935662 29766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:39.935809 29766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:39.936859 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "4d9e1df4190c4fe697980a8b197cb6fe"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "0a20f74af54910598881abb8d2b9a1e0"
server_key_iv: "6cd235c0c5d6f05e6a1c870527c830a2"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.937218 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "4d9e1df4190c4fe697980a8b197cb6fe"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "0a20f74af54910598881abb8d2b9a1e0"
server_key_iv: "6cd235c0c5d6f05e6a1c870527c830a2"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.938611 29766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:15:39.939303 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.939477 29766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:39.939584 29766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "4d9e1df4190c4fe697980a8b197cb6fe"
format_stamp: "Formatted at 2026-08-12 00:15:39 on dist-test-slave-zr1t"
server_key: "0a20f74af54910598881abb8d2b9a1e0"
server_key_iv: "6cd235c0c5d6f05e6a1c870527c830a2"
server_key_version: "encryptionkey@0"
I20260812 00:15:39.939656 29766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:39.994848 29766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:39.996721 29766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:39.996891 29766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:39.997145 29766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:39.997469 29766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:39.997524 29766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:39.997608 29766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:39.997639 29766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.003662 29766 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:36825
I20260812 00:15:40.003708 29973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:36825 every 8 connection(s)
I20260812 00:15:40.004066 29766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:15:40 dist-test-slave-zr1t krb5kdc[29385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:40.013279 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.004786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:34221 (local address 127.31.227.62:46789)
0812 00:15:40.004827 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.004829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.005114 (+ 285us) server_negotiation.cc:407] Connection header received
0812 00:15:40.005722 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.005723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.005737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.005794 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:40.006783 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.007281 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.008402 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.008606 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.008694 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.010948 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:40.010950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:40.010950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:40.010956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:40.012137 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:40.012513 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:40.012514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:40.012515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:40.012565 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:40.012775 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:40.012776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:40.012776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:40.012874 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:40.012923 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.013146 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.013216 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:15:40.013434 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.004775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46789 (local address 127.31.227.2:34221)
0812 00:15:40.004995 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.005029 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.005611 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.005864 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.005869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.006473 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:40.006668 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.006673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.007383 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.007384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.008309 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.008311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.008707 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.008708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.008860 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.009438 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:40.009448 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:40.010853 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:40.012208 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:40.012215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:40.012217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:40.012436 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:40.012622 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:40.012623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:40.012624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:40.012722 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:40.012925 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:40.012929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:40.013041 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.013120 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.013236 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260812 00:15:40.013830 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 29766
I20260812 00:15:40.013952 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:15:40.014077 29974 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46789
I20260812 00:15:40.014160 29974 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:40.014201 32652 external_mini_cluster.cc:1428] Setting key 200add60df633a73a2ab8192f8938bca
I20260812 00:15:40.014356 29974 heartbeater.cc:511] Master 127.31.227.62:46789 requested a full tablet report, sending...
I20260812 00:15:40.014752 29458 ts_manager.cc:194] Registered new tserver with Master: 4d9e1df4190c4fe697980a8b197cb6fe (127.31.227.2:36825)
I20260812 00:15:40.015146 29458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:34221
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:15:40.037993 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46789
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:40.119764 29980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:40.119990 29980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:40.120029 29980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:40.121258 29980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:40.121330 29980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:40.121376 29980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:40.122905 29980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:40.123178 29980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:40.123409 29980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:40.128808 29994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:40.128836 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:40.128820 29995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:40.128865 29980 server_base.cc:1034] running on GCE node
I20260812 00:15:40.129273 29980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:40.129591 29980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:40.130746 29980 hybrid_clock.cc:648] HybridClock initialized: now 1786493740130732 us; error 35 us; skew 500 ppm
Aug 12 00:15:40 dist-test-slave-zr1t krb5kdc[29385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:40.133256 29980 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:15:40.133766 29980 webserver.cc:460] Webserver started at http://127.31.227.3:40055/ using document root <none> and password file <none>
I20260812 00:15:40.134006 29980 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:40.134053 29980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:40.134208 29980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:40.135272 29980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "5bdc858e44cf4f408a965fdafcbd6dbe"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "791e2c1769f47a14d95799a62687a358"
server_key_iv: "37275428a193a3e8c7ea61fdc6484534"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.135628 29980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "5bdc858e44cf4f408a965fdafcbd6dbe"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "791e2c1769f47a14d95799a62687a358"
server_key_iv: "37275428a193a3e8c7ea61fdc6484534"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.137014 29980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:15:40.137787 30011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.137960 29980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:40.138070 29980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "5bdc858e44cf4f408a965fdafcbd6dbe"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "791e2c1769f47a14d95799a62687a358"
server_key_iv: "37275428a193a3e8c7ea61fdc6484534"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.138146 29980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:40.154049 29980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:40.155898 29980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:40.156069 29980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:40.156314 29980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:40.156636 29980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:40.156693 29980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.156735 29980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:40.156762 29980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.163435 29980 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:33553
I20260812 00:15:40.163498 30205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:33553 every 8 connection(s)
I20260812 00:15:40.163800 29980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:15:40 dist-test-slave-zr1t krb5kdc[29385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:40.172614 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 29980
I20260812 00:15:40.172729 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:15:40.172783 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.164523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:53397 (local address 127.31.227.62:46789)
0812 00:15:40.164567 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.164569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.164784 (+ 215us) server_negotiation.cc:407] Connection header received
0812 00:15:40.165264 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.165265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.165274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.165331 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:40.166311 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.166876 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.167822 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.168006 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.168056 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.170161 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:40.170163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:40.170163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:40.170169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:40.171581 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:40.171923 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:40.171924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:40.171925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:40.171975 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:40.172181 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:40.172182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:40.172182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:40.172281 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:40.172330 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.172578 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.172662 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:15:40.172946 32652 external_mini_cluster.cc:1428] Setting key 5334063d43de503ef37db38c0cad8972
I20260812 00:15:40.172962 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.164488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:46789 (local address 127.31.227.3:53397)
0812 00:15:40.164652 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.164694 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.165183 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.165385 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.165388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.166034 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:40.166228 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.166233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.166969 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.166969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.167750 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.167751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.168079 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.168080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.168186 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.168632 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:40.168640 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:40.170074 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:40.171651 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:40.171656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:40.171658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:40.171861 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:40.172032 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:40.172033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:40.172033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:40.172129 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:40.172334 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:40.172339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:40.172449 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.172528 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.172643 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260812 00:15:40.173488 30206 heartbeater.cc:347] Connected to a master server at 127.31.227.62:46789
I20260812 00:15:40.173659 30206 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:40.173851 30206 heartbeater.cc:511] Master 127.31.227.62:46789 requested a full tablet report, sending...
I20260812 00:15:40.174242 29458 ts_manager.cc:194] Registered new tserver with Master: 5bdc858e44cf4f408a965fdafcbd6dbe (127.31.227.3:33553)
I20260812 00:15:40.174583 29458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:53397
I20260812 00:15:40.184809 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:40.191890 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.186230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.31.227.62:46789)
0812 00:15:40.186292 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.186294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.186421 (+ 127us) server_negotiation.cc:407] Connection header received
0812 00:15:40.186471 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.186472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.186481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.186531 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:40.187064 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.187529 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.188225 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.188394 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.188441 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.189513 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:40.189555 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:40.189556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:40.189563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:40.190961 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:40.191254 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:40.191255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:40.191255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:40.191307 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:40.191449 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:40.191449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:40.191450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:40.191582 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:40.191654 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.191747 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.191802 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:15:40.193792 29458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:40.195295 29458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:40.201859 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.197595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:33553 (local address 127.0.0.1:42362)
0812 00:15:40.197724 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.197768 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.197835 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.198174 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.198177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.198180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:40.198261 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.198264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.199575 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.199576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.200649 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.200652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.201508 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.201509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.201623 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.201638 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.201719 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.201777 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260812 00:15:40.201866 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.197751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.31.227.3:33553)
0812 00:15:40.198001 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.198007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.198017 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:40.198058 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.198060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.198075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.198158 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:40.198334 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.199424 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.200736 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.201396 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.201479 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.201559 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.201744 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.201788 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260812 00:15:40.202293 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.197956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.31.227.2:36825)
0812 00:15:40.198154 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.198159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.198276 (+ 117us) server_negotiation.cc:407] Connection header received
0812 00:15:40.198321 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.198323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.198339 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.198411 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:40.198634 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.199652 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.201072 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.201693 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.201754 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.201808 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.202149 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.202199 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260812 00:15:40.202085 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.197642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:36825 (local address 127.0.0.1:37838)
0812 00:15:40.198141 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.198153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.198219 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.198488 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.198489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.198491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:40.198552 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.198554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.199796 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.199797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.200960 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.200964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.201842 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.201843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.201909 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.201913 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.201969 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.202016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":98,"threads_started":1}
I20260812 00:15:40.203014 29874 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2cbe97cd314051b35067a63f651e22 (DEFAULT_TABLE table=test-table [id=6987e79e2682415abcaf66e23c114f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:40.203023 30092 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2cbe97cd314051b35067a63f651e22 (DEFAULT_TABLE table=test-table [id=6987e79e2682415abcaf66e23c114f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:40.203390 30092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2cbe97cd314051b35067a63f651e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:40.203390 29874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2cbe97cd314051b35067a63f651e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:40.203614 30234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.197808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41883 (local address 127.0.0.1:37964)
0812 00:15:40.198259 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.198269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.198320 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.199755 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.199755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.199758 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:40.199802 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.199804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.201032 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.201033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.202067 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.202070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.203366 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.203367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.203437 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.203442 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.203500 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.203553 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":65,"threads_started":1}
I20260812 00:15:40.203902 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.197913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.31.227.1:41883)
0812 00:15:40.198244 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.198247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.199592 (+ 1345us) server_negotiation.cc:407] Connection header received
0812 00:15:40.199643 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.199645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.199662 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.199745 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:40.199868 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.200919 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.202162 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.203267 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.203344 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.203402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.203771 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.203818 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260812 00:15:40.204497 29654 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2cbe97cd314051b35067a63f651e22 (DEFAULT_TABLE table=test-table [id=6987e79e2682415abcaf66e23c114f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:40.204835 29654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2cbe97cd314051b35067a63f651e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:40.206436 30237 tablet_bootstrap.cc:492] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Bootstrap starting.
I20260812 00:15:40.207000 30239 tablet_bootstrap.cc:492] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Bootstrap starting.
I20260812 00:15:40.207165 30237 tablet_bootstrap.cc:654] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:40.207784 30239 tablet_bootstrap.cc:654] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:40.207835 30237 log.cc:824] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:40.208173 30242 tablet_bootstrap.cc:492] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: Bootstrap starting.
I20260812 00:15:40.208418 30239 log.cc:824] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:40.208813 30242 tablet_bootstrap.cc:654] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:40.209276 30239 tablet_bootstrap.cc:492] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: No bootstrap required, opened a new log
I20260812 00:15:40.209358 30239 ts_tablet_manager.cc:1397] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:40.209556 30242 log.cc:824] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:40.210140 30237 tablet_bootstrap.cc:492] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: No bootstrap required, opened a new log
I20260812 00:15:40.210217 30237 ts_tablet_manager.cc:1397] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 00:15:40.211468 30239 raft_consensus.cc:348] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.211691 30239 raft_consensus.cc:374] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:40.211730 30239 raft_consensus.cc:729] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9e1df4190c4fe697980a8b197cb6fe, State: Initialized, Role: FOLLOWER
I20260812 00:15:40.211850 30239 consensus_queue.cc:260] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.212173 30239 ts_tablet_manager.cc:1428] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:40.212188 29974 heartbeater.cc:503] Master 127.31.227.62:46789 was elected leader, sending a full tablet report...
I20260812 00:15:40.212297 30237 raft_consensus.cc:348] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.212472 30237 raft_consensus.cc:374] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:40.212518 30237 raft_consensus.cc:729] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdc858e44cf4f408a965fdafcbd6dbe, State: Initialized, Role: FOLLOWER
I20260812 00:15:40.212625 30237 consensus_queue.cc:260] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.212889 30237 ts_tablet_manager.cc:1428] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:40.212921 30206 heartbeater.cc:503] Master 127.31.227.62:46789 was elected leader, sending a full tablet report...
I20260812 00:15:40.213641 30242 tablet_bootstrap.cc:492] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: No bootstrap required, opened a new log
I20260812 00:15:40.213721 30242 ts_tablet_manager.cc:1397] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 00:15:40.215291 30242 raft_consensus.cc:348] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.215407 30242 raft_consensus.cc:374] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:40.215426 30242 raft_consensus.cc:729] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57aeaba97934431fa401ddb34158b386, State: Initialized, Role: FOLLOWER
I20260812 00:15:40.215536 30242 consensus_queue.cc:260] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.215762 30242 ts_tablet_manager.cc:1428] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:40.215835 29758 heartbeater.cc:503] Master 127.31.227.62:46789 was elected leader, sending a full tablet report...
W20260812 00:15:40.254781 29975 tablet.cc:2369] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:40.306344 29759 tablet.cc:2369] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:40.414506 30209 tablet.cc:2369] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:15:40.528839 30249 raft_consensus.cc:482] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:40.528965 30249 raft_consensus.cc:504] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.529373 30249 leader_election.cc:288] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bdc858e44cf4f408a965fdafcbd6dbe (127.31.227.3:33553), 57aeaba97934431fa401ddb34158b386 (127.31.227.1:41883)
I20260812 00:15:40.532506 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.529900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:47993 (local address 127.31.227.1:41883)
0812 00:15:40.529952 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.529954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.529987 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:15:40.530087 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.530087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.530096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.530154 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:40.530349 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.530849 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.531817 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.532156 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.532211 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.532218 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.532393 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.532436 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:15:40.532536 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.529820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41883 (local address 127.31.227.2:47993)
0812 00:15:40.529927 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.529937 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.530004 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.530186 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.530187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.530189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:40.530280 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.530282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.530939 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.530940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.531696 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.531698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.532241 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.532243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.532314 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.532318 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.532369 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.532401 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260812 00:15:40.532575 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.529696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:39973 (local address 127.31.227.3:33553)
0812 00:15:40.529765 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.529768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.529834 (+ 66us) server_negotiation.cc:407] Connection header received
0812 00:15:40.530006 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.530008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.530020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.530076 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:40.530255 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.530788 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.531785 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.532182 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.532275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.532282 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.532459 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.532501 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:15:40.532830 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.529607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:33553 (local address 127.31.227.2:39973)
0812 00:15:40.529761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.529775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.529855 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.530090 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.530091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.530094 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:40.530176 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.530179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.530875 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.530876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.531663 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.531666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.532293 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.532294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.532375 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.532379 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.532436 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.532471 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":11,"thread_start_us":83,"threads_started":1}
I20260812 00:15:40.533010 29682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2cbe97cd314051b35067a63f651e22" candidate_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57aeaba97934431fa401ddb34158b386" is_pre_election: true
I20260812 00:15:40.533174 29682 raft_consensus.cc:2454] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d9e1df4190c4fe697980a8b197cb6fe in term 0.
I20260812 00:15:40.533201 30127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2cbe97cd314051b35067a63f651e22" candidate_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" is_pre_election: true
I20260812 00:15:40.533384 30127 raft_consensus.cc:2454] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d9e1df4190c4fe697980a8b197cb6fe in term 0.
I20260812 00:15:40.533418 29796 leader_election.cc:302] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d9e1df4190c4fe697980a8b197cb6fe, 57aeaba97934431fa401ddb34158b386; no voters:
I20260812 00:15:40.533586 30249 raft_consensus.cc:2781] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:40.533656 30249 raft_consensus.cc:482] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:40.533674 30249 raft_consensus.cc:3037] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:40.534476 30249 raft_consensus.cc:504] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.534601 30249 leader_election.cc:288] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [CANDIDATE]: Term 1 election: Requested vote from peers 5bdc858e44cf4f408a965fdafcbd6dbe (127.31.227.3:33553), 57aeaba97934431fa401ddb34158b386 (127.31.227.1:41883)
I20260812 00:15:40.534785 29682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2cbe97cd314051b35067a63f651e22" candidate_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57aeaba97934431fa401ddb34158b386"
I20260812 00:15:40.534785 30127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2cbe97cd314051b35067a63f651e22" candidate_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe"
I20260812 00:15:40.534866 30127 raft_consensus.cc:3037] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:40.534866 29682 raft_consensus.cc:3037] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:40.535758 29682 raft_consensus.cc:2454] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d9e1df4190c4fe697980a8b197cb6fe in term 1.
I20260812 00:15:40.535758 30127 raft_consensus.cc:2454] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d9e1df4190c4fe697980a8b197cb6fe in term 1.
I20260812 00:15:40.535952 29796 leader_election.cc:302] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d9e1df4190c4fe697980a8b197cb6fe, 5bdc858e44cf4f408a965fdafcbd6dbe; no voters:
I20260812 00:15:40.536042 30249 raft_consensus.cc:2781] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:40.536192 30249 raft_consensus.cc:686] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d9e1df4190c4fe697980a8b197cb6fe, State: Running, Role: LEADER
I20260812 00:15:40.536299 30249 consensus_queue.cc:237] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } }
I20260812 00:15:40.537134 29459 catalog_manager.cc:5495] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d9e1df4190c4fe697980a8b197cb6fe (127.31.227.2). New cstate: current_term: 1 leader_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d9e1df4190c4fe697980a8b197cb6fe" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260812 00:15:40.553442 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.551081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.31.227.2:36825)
0812 00:15:40.551138 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.551140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.551150 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:40.551226 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.551227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.551251 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.551327 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:40.551485 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.551926 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.552701 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.552884 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.552971 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.553158 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:40.553247 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:40.553334 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.553357 (+ 23us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.553375 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:15:40.555234 30127 raft_consensus.cc:1261] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 1 FOLLOWER]: Refusing update from remote peer 4d9e1df4190c4fe697980a8b197cb6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:40.555234 29682 raft_consensus.cc:1261] T 9c2cbe97cd314051b35067a63f651e22 P 57aeaba97934431fa401ddb34158b386 [term 1 FOLLOWER]: Refusing update from remote peer 4d9e1df4190c4fe697980a8b197cb6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:40.555569 30249 consensus_queue.cc:1035] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bdc858e44cf4f408a965fdafcbd6dbe" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:40.555738 30265 consensus_queue.cc:1035] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "57aeaba97934431fa401ddb34158b386" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:40.558285 30269 mvcc.cc:205] Tried to move back new op lower bound from 7317478361311318016 to 7317478361237372928. Current Snapshot: MvccSnapshot[applied={T|T < 7317478361311318016 or (T in {7317478361311318016})}]
I20260812 00:15:40.558418 30272 mvcc.cc:205] Tried to move back new op lower bound from 7317478361311318016 to 7317478361237372928. Current Snapshot: MvccSnapshot[applied={T|T < 7317478361311318016}]
I20260812 00:15:40.559090 30279 mvcc.cc:205] Tried to move back new op lower bound from 7317478361311318016 to 7317478361237372928. Current Snapshot: MvccSnapshot[applied={T|T < 7317478361311318016}]
I20260812 00:15:40.560421 29457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:40.560537 29457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:40.561308 29457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fffc0184d7b44f38917a615e50090f00: Sending DeleteTablet for 3 replicas of tablet 9c2cbe97cd314051b35067a63f651e22
I20260812 00:15:40.561668 29874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2cbe97cd314051b35067a63f651e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:40 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:37838
I20260812 00:15:40.561734 30092 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2cbe97cd314051b35067a63f651e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:40 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:42362
I20260812 00:15:40.561966 30291 tablet_replica.cc:330] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: stopping tablet replica
I20260812 00:15:40.561970 29654 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2cbe97cd314051b35067a63f651e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:40 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:37964
I20260812 00:15:40.562104 30291 raft_consensus.cc:2229] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:40.562121 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 29553
I20260812 00:15:40.562207 30291 raft_consensus.cc:2258] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:40.562747 30290 tablet_replica.cc:330] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: stopping tablet replica
I20260812 00:15:40.562831 30290 raft_consensus.cc:2229] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:15:40.562915 30290 raft_consensus.cc:2258] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:40.566781 30290 ts_tablet_manager.cc:1905] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:40.567934 30291 ts_tablet_manager.cc:1905] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:40.568542 30290 ts_tablet_manager.cc:1918] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:40.568614 30290 log.cc:1196] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/9c2cbe97cd314051b35067a63f651e22
I20260812 00:15:40.568864 30290 ts_tablet_manager.cc:1939] T 9c2cbe97cd314051b35067a63f651e22 P 4d9e1df4190c4fe697980a8b197cb6fe: Deleting consensus metadata
I20260812 00:15:40.569267 29429 catalog_manager.cc:4841] TS 4d9e1df4190c4fe697980a8b197cb6fe (127.31.227.2:36825): tablet 9c2cbe97cd314051b35067a63f651e22 (table test-table [id=6987e79e2682415abcaf66e23c114f7d]) successfully deleted
I20260812 00:15:40.570138 30291 ts_tablet_manager.cc:1918] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:40.570194 30291 log.cc:1196] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/9c2cbe97cd314051b35067a63f651e22
I20260812 00:15:40.570392 30291 ts_tablet_manager.cc:1939] T 9c2cbe97cd314051b35067a63f651e22 P 5bdc858e44cf4f408a965fdafcbd6dbe: Deleting consensus metadata
I20260812 00:15:40.570808 29427 catalog_manager.cc:4841] TS 5bdc858e44cf4f408a965fdafcbd6dbe (127.31.227.3:33553): tablet 9c2cbe97cd314051b35067a63f651e22 (table test-table [id=6987e79e2682415abcaf66e23c114f7d]) successfully deleted
W20260812 00:15:40.580935 29427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:41883 (error 108)
I20260812 00:15:40.581190 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 29766
I20260812 00:15:40.581830 30234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.581365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41883 (local address 127.0.0.1:55248)
0812 00:15:40.581714 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.581774 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
W20260812 00:15:40.584625 29427 catalog_manager.cc:4584] TS 57aeaba97934431fa401ddb34158b386 (127.31.227.1:41883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c2cbe97cd314051b35067a63f651e22: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41883: connect: Connection refused (error 111)
I20260812 00:15:40.590709 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 29980
I20260812 00:15:40.605702 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 29401
2026-08-12T00:15:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2520 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T00:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:40Z Disabled control of system clock
I20260812 00:15:40.638211 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34101
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:40.720338 30302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:40.720551 30302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:40.720572 30302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:40.721705 30302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:40.721755 30302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:40.721767 30302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:40.721779 30302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:40.721791 30302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:40.723183 30302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:40.723390 30302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:40.723624 30302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:40.728823 30310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:40.728933 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:40.728895 30302 server_base.cc:1034] running on GCE node
W20260812 00:15:40.728837 30315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:40.729334 30302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:40.729698 30302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:40.730896 30302 hybrid_clock.cc:648] HybridClock initialized: now 1786493740730878 us; error 44 us; skew 500 ppm
I20260812 00:15:40.732197 30302 webserver.cc:460] Webserver started at http://127.31.227.62:33255/ using document root <none> and password file <none>
I20260812 00:15:40.732478 30302 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:40.732553 30302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:40.732717 30302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:40.733870 30302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "378f55bc30944cafb9be6650dff7d8fd"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "0abc75381120d53bac52038c0aa32297"
server_key_iv: "d43881c9563965c7ccc26f523da179a5"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.734242 30302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "378f55bc30944cafb9be6650dff7d8fd"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "0abc75381120d53bac52038c0aa32297"
server_key_iv: "d43881c9563965c7ccc26f523da179a5"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.735540 30302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:40.736253 30321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.736436 30302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:40.736552 30302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "378f55bc30944cafb9be6650dff7d8fd"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "0abc75381120d53bac52038c0aa32297"
server_key_iv: "d43881c9563965c7ccc26f523da179a5"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.736627 30302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:40.745677 30302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:40.745980 30302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:40.746138 30302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:40.750174 30302 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34101
I20260812 00:15:40.750206 30403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34101 every 8 connection(s)
I20260812 00:15:40.750560 30302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:40.751521 30405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:40.752734 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 30302
I20260812 00:15:40.752859 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:40.753065 32652 external_mini_cluster.cc:1428] Setting key 20965f123b0aff11867829a6208908bd
I20260812 00:15:40.754235 30405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd: Bootstrap starting.
I20260812 00:15:40.754806 30405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:40.755105 30405 log.cc:824] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:40.755820 30405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd: No bootstrap required, opened a new log
I20260812 00:15:40.757565 30405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f55bc30944cafb9be6650dff7d8fd" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34101 } }
I20260812 00:15:40.757751 30405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:40.757799 30405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378f55bc30944cafb9be6650dff7d8fd, State: Initialized, Role: FOLLOWER
I20260812 00:15:40.757949 30405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f55bc30944cafb9be6650dff7d8fd" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34101 } }
I20260812 00:15:40.758056 30405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:40.758093 30405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:40.758142 30405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:40.758646 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.753642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.31.227.62:34101)
0812 00:15:40.753869 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.753876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.753907 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:15:40.754305 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.754313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.754395 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.754538 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:40.754721 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.755844 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.756611 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.757017 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.757148 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.757899 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:40.757903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:40.757903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:40.758118 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:40.758162 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:40.758178 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:40.758265 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.758332 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.758413 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260812 00:15:40.758826 30405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f55bc30944cafb9be6650dff7d8fd" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34101 } }
I20260812 00:15:40.758908 30405 leader_election.cc:302] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 378f55bc30944cafb9be6650dff7d8fd; no voters:
I20260812 00:15:40.759075 30405 leader_election.cc:288] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:40.759152 30415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:40.759315 30405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:40.759338 30415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [term 1 LEADER]: Becoming Leader. State: Replica: 378f55bc30944cafb9be6650dff7d8fd, State: Running, Role: LEADER
I20260812 00:15:40.759521 30415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f55bc30944cafb9be6650dff7d8fd" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34101 } }
I20260812 00:15:40.759902 30418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 378f55bc30944cafb9be6650dff7d8fd. Latest consensus state: current_term: 1 leader_uuid: "378f55bc30944cafb9be6650dff7d8fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f55bc30944cafb9be6650dff7d8fd" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34101 } } }
I20260812 00:15:40.759982 30418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:40.760324 30416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "378f55bc30944cafb9be6650dff7d8fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f55bc30944cafb9be6650dff7d8fd" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34101 } } }
I20260812 00:15:40.760351 30433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:40.760421 30416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:40.760934 30433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:40.762599 30433 catalog_manager.cc:1347] Generated new cluster ID: c025c52d59284edba46621c37dd9174c
I20260812 00:15:40.762657 30433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:40.773964 30433 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:40.774652 30433 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:40.777819 30433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd: Generated new TSK 0
I20260812 00:15:40.778005 30433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:40.779702 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34101
--builtin_ntp_servers=127.31.227.20:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:40.882802 30452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:40.882999 30452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:40.883023 30452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:40.884148 30452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:40.884188 30452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:40.884213 30452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:40.885612 30452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:40.885840 30452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:40.886142 30452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:40.891343 30463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:40.891405 30452 server_base.cc:1034] running on GCE node
W20260812 00:15:40.891351 30465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:40.891700 30472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:40.891909 30452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:40.892208 30452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:40.893364 30452 hybrid_clock.cc:648] HybridClock initialized: now 1786493740893352 us; error 28 us; skew 500 ppm
I20260812 00:15:40.894683 30452 webserver.cc:460] Webserver started at http://127.31.227.1:45513/ using document root <none> and password file <none>
I20260812 00:15:40.894913 30452 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:40.894985 30452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:40.895116 30452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:40.896173 30452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "c990dba04a0244a7a5b43de9d63a7408"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "154fc6c4609cac36147d0540f40d5c8b"
server_key_iv: "e26c86fd687e99c1ce4c7e815a36d4f6"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.896549 30452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "c990dba04a0244a7a5b43de9d63a7408"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "154fc6c4609cac36147d0540f40d5c8b"
server_key_iv: "e26c86fd687e99c1ce4c7e815a36d4f6"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.897933 30452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 00:15:40.898705 30490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.898970 30452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:40.899076 30452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "c990dba04a0244a7a5b43de9d63a7408"
format_stamp: "Formatted at 2026-08-12 00:15:40 on dist-test-slave-zr1t"
server_key: "154fc6c4609cac36147d0540f40d5c8b"
server_key_iv: "e26c86fd687e99c1ce4c7e815a36d4f6"
server_key_version: "encryptionkey@0"
I20260812 00:15:40.899154 30452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:40.922356 30452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:40.922684 30452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:40.922818 30452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:40.923089 30452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:40.923472 30452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:40.923534 30452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.923574 30452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:40.923599 30452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:40.930595 30452 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:41917
I20260812 00:15:40.930653 30680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:41917 every 8 connection(s)
I20260812 00:15:40.930999 30452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:15:40.936039 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.931773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:38595 (local address 127.31.227.62:34101)
0812 00:15:40.931832 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:40.931835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:40.932089 (+ 254us) server_negotiation.cc:407] Connection header received
0812 00:15:40.932584 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:40.932585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:40.932597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:40.932647 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:40.933288 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.933843 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.934772 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.934957 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.935005 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.935829 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:40.935830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:40.935830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:40.935834 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:40.935844 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:40.935847 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:40.935884 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:40.935965 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 00:15:40.935983 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:15:40.936272 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:40.931778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34101 (local address 127.31.227.1:38595)
0812 00:15:40.931967 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:40.932001 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:40.932502 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:40.932707 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:40.932710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:40.932718 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:40.933221 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.933225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.933923 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.933924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.934700 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:40.934702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:40.935028 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:40.935028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:40.935133 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:40.935708 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:40.935709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:40.935736 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:40.935759 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:40.935900 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:40.935902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:40.935904 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:15:40.935951 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 00:15:40.936028 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260812 00:15:40.936630 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 30452
I20260812 00:15:40.936746 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:15:40.936828 30681 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34101
I20260812 00:15:40.936940 30681 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:40.936964 32652 external_mini_cluster.cc:1428] Setting key 3f65ecee4ab6861c3e572f6ade2776a1
I20260812 00:15:40.937147 30681 heartbeater.cc:511] Master 127.31.227.62:34101 requested a full tablet report, sending...
I20260812 00:15:40.937750 30344 ts_manager.cc:194] Registered new tserver with Master: c990dba04a0244a7a5b43de9d63a7408 (127.31.227.1:41917)
I20260812 00:15:40.937971 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34101
--builtin_ntp_servers=127.31.227.20:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 00:15:40.938637 30344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.1:38595
W20260812 00:15:41.020274 30688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:41.020498 30688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:41.020533 30688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:41.021709 30688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:41.021765 30688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:41.021807 30688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:41.023156 30688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:41.023420 30688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:41.023720 30688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:41.028762 30698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:41.028925 30694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:41.028925 30693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:41.029251 30688 server_base.cc:1034] running on GCE node
I20260812 00:15:41.029503 30688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:41.029804 30688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:41.030983 30688 hybrid_clock.cc:648] HybridClock initialized: now 1786493741030965 us; error 32 us; skew 500 ppm
I20260812 00:15:41.032416 30688 webserver.cc:460] Webserver started at http://127.31.227.2:36165/ using document root <none> and password file <none>
I20260812 00:15:41.032693 30688 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:41.032764 30688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:41.032923 30688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:41.034092 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "b8c2d536dbb34ad69424628110844a1e"
format_stamp: "Formatted at 2026-08-12 00:15:41 on dist-test-slave-zr1t"
server_key: "54dd814d2e66838942bbc3ece31d2e3f"
server_key_iv: "cce171f769d61157b5d88060e931b212"
server_key_version: "encryptionkey@0"
I20260812 00:15:41.034471 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "b8c2d536dbb34ad69424628110844a1e"
format_stamp: "Formatted at 2026-08-12 00:15:41 on dist-test-slave-zr1t"
server_key: "54dd814d2e66838942bbc3ece31d2e3f"
server_key_iv: "cce171f769d61157b5d88060e931b212"
server_key_version: "encryptionkey@0"
I20260812 00:15:41.035799 30688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:15:41.036576 30709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:41.036746 30688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:41.036849 30688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "b8c2d536dbb34ad69424628110844a1e"
format_stamp: "Formatted at 2026-08-12 00:15:41 on dist-test-slave-zr1t"
server_key: "54dd814d2e66838942bbc3ece31d2e3f"
server_key_iv: "cce171f769d61157b5d88060e931b212"
server_key_version: "encryptionkey@0"
I20260812 00:15:41.036936 30688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:41.049448 30688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:41.049780 30688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:41.049945 30688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:41.050165 30688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:41.050500 30688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:41.050570 30688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:41.050606 30688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:41.050632 30688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:41.056691 30688 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:41445
I20260812 00:15:41.056746 30912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:41445 every 8 connection(s)
I20260812 00:15:41.057060 30688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:15:41.062207 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.057752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:38253 (local address 127.31.227.62:34101)
0812 00:15:41.057813 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.057815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.058056 (+ 241us) server_negotiation.cc:407] Connection header received
0812 00:15:41.058545 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.058546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.058556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.058607 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:41.059219 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.059756 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.060912 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.061119 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.061193 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.061999 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:41.062000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:41.062000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:41.062005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:41.062019 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:41.062022 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:41.062064 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.062130 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.062150 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:15:41.062458 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.057758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34101 (local address 127.31.227.2:38253)
0812 00:15:41.057953 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.057985 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.058460 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.058657 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.058660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.058668 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:41.059151 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.059155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.059835 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.059836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.060838 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.060841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.061208 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.061209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.061316 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.061836 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:41.061837 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:41.061867 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:41.061907 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:41.062077 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:41.062082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:41.062084 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.062133 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.062207 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260812 00:15:41.062927 30913 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34101
I20260812 00:15:41.063017 30913 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:41.063211 30913 heartbeater.cc:511] Master 127.31.227.62:34101 requested a full tablet report, sending...
I20260812 00:15:41.063585 30344 ts_manager.cc:194] Registered new tserver with Master: b8c2d536dbb34ad69424628110844a1e (127.31.227.2:41445)
I20260812 00:15:41.063957 30344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.2:38253
I20260812 00:15:41.064011 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 30688
I20260812 00:15:41.064157 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:15:41.064400 32652 external_mini_cluster.cc:1428] Setting key 7ef7ab67044ca9a36891e9c6c9370415
I20260812 00:15:41.065402 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34101
--builtin_ntp_servers=127.31.227.20:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:41.148473 30919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:41.148720 30919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:41.148753 30919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:41.149919 30919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:41.149983 30919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:41.150024 30919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:41.151418 30919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:34101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:41.151686 30919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:41.151999 30919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:41.157135 30934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:41.157229 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:41.157146 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:41.157244 30919 server_base.cc:1034] running on GCE node
I20260812 00:15:41.157604 30919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:41.157860 30919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:41.158999 30919 hybrid_clock.cc:648] HybridClock initialized: now 1786493741158987 us; error 29 us; skew 500 ppm
I20260812 00:15:41.160518 30919 webserver.cc:460] Webserver started at http://127.31.227.3:43739/ using document root <none> and password file <none>
I20260812 00:15:41.160766 30919 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:41.160913 30919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:41.161087 30919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:41.162199 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "00dd182ab40843dd9e5f210aea6300d9"
format_stamp: "Formatted at 2026-08-12 00:15:41 on dist-test-slave-zr1t"
server_key: "f9a475f7bbb2d6543707ddf889880f07"
server_key_iv: "cc7890d33b64b73d5c3ae9cfbd3b9f34"
server_key_version: "encryptionkey@0"
I20260812 00:15:41.162524 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "00dd182ab40843dd9e5f210aea6300d9"
format_stamp: "Formatted at 2026-08-12 00:15:41 on dist-test-slave-zr1t"
server_key: "f9a475f7bbb2d6543707ddf889880f07"
server_key_iv: "cc7890d33b64b73d5c3ae9cfbd3b9f34"
server_key_version: "encryptionkey@0"
I20260812 00:15:41.163774 30919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:41.164527 30941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:41.164693 30919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:41.164767 30919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "00dd182ab40843dd9e5f210aea6300d9"
format_stamp: "Formatted at 2026-08-12 00:15:41 on dist-test-slave-zr1t"
server_key: "f9a475f7bbb2d6543707ddf889880f07"
server_key_iv: "cc7890d33b64b73d5c3ae9cfbd3b9f34"
server_key_version: "encryptionkey@0"
I20260812 00:15:41.164819 30919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:41.181664 30919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:41.181902 30919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:41.182004 30919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:41.182176 30919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:41.182459 30919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:41.182490 30919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:41.182511 30919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:41.182523 30919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:41.188346 30919 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:43853
I20260812 00:15:41.188395 31118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:43853 every 8 connection(s)
I20260812 00:15:41.188709 30919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:15:41.191375 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 30919
I20260812 00:15:41.191496 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:15:41.191716 32652 external_mini_cluster.cc:1428] Setting key d38e5fdd9198fc7e1d2df7d2a3a2252d
I20260812 00:15:41.193948 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.189381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:38011 (local address 127.31.227.62:34101)
0812 00:15:41.189471 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.189473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.189723 (+ 250us) server_negotiation.cc:407] Connection header received
0812 00:15:41.190168 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.190169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.190181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.190229 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:41.190867 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.191364 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.192570 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.192840 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.192893 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.193731 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:41.193732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:41.193732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:41.193736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:41.193751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:41.193754 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:41.193805 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.193869 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.193889 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 00:15:41.194140 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.189347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34101 (local address 127.31.227.3:38011)
0812 00:15:41.189595 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.189637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.190085 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.190284 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.190287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.190295 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:41.190799 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.190803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.191476 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.191477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.192490 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.192492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.192915 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.192916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.193022 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.193564 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:41.193565 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:41.193604 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:41.193627 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:41.193810 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:41.193815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:41.193818 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.193864 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.193938 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260812 00:15:41.194589 31119 heartbeater.cc:347] Connected to a master server at 127.31.227.62:34101
I20260812 00:15:41.194697 31119 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:41.194877 31119 heartbeater.cc:511] Master 127.31.227.62:34101 requested a full tablet report, sending...
I20260812 00:15:41.195258 30344 ts_manager.cc:194] Registered new tserver with Master: 00dd182ab40843dd9e5f210aea6300d9 (127.31.227.3:43853)
I20260812 00:15:41.195632 30344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.3:38011
I20260812 00:15:41.203542 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:41.207653 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.204900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.31.227.62:34101)
0812 00:15:41.204972 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.204974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.205064 (+ 90us) server_negotiation.cc:407] Connection header received
0812 00:15:41.205097 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.205098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.205107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.205154 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:41.205308 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.205863 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.206564 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.206737 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.206786 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.207380 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:41.207382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:41.207382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:41.207386 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:41.207399 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:41.207403 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:41.207446 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.207518 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.207537 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:15:41.209321 30344 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:41.210906 30344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:41.217311 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.213366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.31.227.1:41917)
0812 00:15:41.213609 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.213613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.213674 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:15:41.213736 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.213738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.213752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.213825 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:41.213995 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.215159 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.216419 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.216915 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.217003 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.217095 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.217187 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.217224 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260812 00:15:41.217299 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.213090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41917 (local address 127.0.0.1:46556)
0812 00:15:41.213623 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.213631 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.213679 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.213831 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.213832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.213837 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:41.213898 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.213900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.215242 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.215243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.216348 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.216350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.217006 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.217007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.217083 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.217097 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.217172 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.217225 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":44,"thread_start_us":100,"threads_started":1}
I20260812 00:15:41.217486 31130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.213080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:41445 (local address 127.0.0.1:38496)
0812 00:15:41.213453 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.213463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.213503 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.213828 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.213831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.213833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:41.213964 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.213966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.215247 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.215248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.216148 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.216150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.217198 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.217198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.217264 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.217271 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.217321 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.217372 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":11,"thread_start_us":59,"threads_started":1}
I20260812 00:15:41.217737 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.213264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.31.227.2:41445)
0812 00:15:41.213626 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.213629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.213638 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:41.213687 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.213689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.213706 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.213800 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:41.214052 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.215139 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.216243 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.216990 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.217069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.217157 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.217611 (+ 454us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.217654 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":96,"threads_started":1}
I20260812 00:15:41.218030 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.213080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43853 (local address 127.0.0.1:39618)
0812 00:15:41.213358 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.213400 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.213485 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.214030 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.214031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.214033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:41.214082 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.214083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.214987 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.214989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.216259 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.216262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.217196 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.217198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.217264 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.217270 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.217321 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.217739 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":391,"thread_start_us":96,"threads_started":1}
I20260812 00:15:41.218253 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.213247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.31.227.3:43853)
0812 00:15:41.213459 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.213464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.213476 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:15:41.213912 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.213914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.213929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.214161 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:41.214217 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.214891 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.216346 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.217018 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.217074 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.217157 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.218134 (+ 977us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.218178 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260812 00:15:41.218129 30562 tablet_service.cc:1467] Processing CreateTablet for tablet 42d2771c61a3474ca99bfd3465c16463 (DEFAULT_TABLE table=test-table [id=2bdadbcbfddd47e1b95a35677c39e6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:41.218384 30796 tablet_service.cc:1467] Processing CreateTablet for tablet 42d2771c61a3474ca99bfd3465c16463 (DEFAULT_TABLE table=test-table [id=2bdadbcbfddd47e1b95a35677c39e6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:41.218576 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d2771c61a3474ca99bfd3465c16463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:41.218721 30796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d2771c61a3474ca99bfd3465c16463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:41.221149 31141 tablet_bootstrap.cc:492] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: Bootstrap starting.
I20260812 00:15:41.221903 31141 tablet_bootstrap.cc:654] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:41.221909 31011 tablet_service.cc:1467] Processing CreateTablet for tablet 42d2771c61a3474ca99bfd3465c16463 (DEFAULT_TABLE table=test-table [id=2bdadbcbfddd47e1b95a35677c39e6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:41.222249 31141 log.cc:824] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:41.222265 31011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d2771c61a3474ca99bfd3465c16463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:41.222622 31138 tablet_bootstrap.cc:492] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Bootstrap starting.
I20260812 00:15:41.223188 31138 tablet_bootstrap.cc:654] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:41.223477 31138 log.cc:824] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:41.224031 31141 tablet_bootstrap.cc:492] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: No bootstrap required, opened a new log
I20260812 00:15:41.224116 31141 ts_tablet_manager.cc:1397] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260812 00:15:41.224231 31138 tablet_bootstrap.cc:492] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: No bootstrap required, opened a new log
I20260812 00:15:41.224301 31138 ts_tablet_manager.cc:1397] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:41.226315 31141 raft_consensus.cc:348] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.226501 31141 raft_consensus.cc:374] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:41.226550 31141 raft_consensus.cc:729] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c990dba04a0244a7a5b43de9d63a7408, State: Initialized, Role: FOLLOWER
I20260812 00:15:41.226675 31141 consensus_queue.cc:260] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.226645 31138 raft_consensus.cc:348] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.226775 31138 raft_consensus.cc:374] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:41.226828 31138 raft_consensus.cc:729] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d536dbb34ad69424628110844a1e, State: Initialized, Role: FOLLOWER
I20260812 00:15:41.226959 31141 ts_tablet_manager.cc:1428] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 00:15:41.226934 31138 consensus_queue.cc:260] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.227007 30681 heartbeater.cc:503] Master 127.31.227.62:34101 was elected leader, sending a full tablet report...
I20260812 00:15:41.227175 31138 ts_tablet_manager.cc:1428] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:41.227169 31146 tablet_bootstrap.cc:492] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Bootstrap starting.
I20260812 00:15:41.227298 30913 heartbeater.cc:503] Master 127.31.227.62:34101 was elected leader, sending a full tablet report...
I20260812 00:15:41.227909 31146 tablet_bootstrap.cc:654] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:41.228242 31146 log.cc:824] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:41.229070 31146 tablet_bootstrap.cc:492] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: No bootstrap required, opened a new log
I20260812 00:15:41.229123 31146 ts_tablet_manager.cc:1397] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:41.230978 31146 raft_consensus.cc:348] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.231097 31146 raft_consensus.cc:374] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:41.231118 31146 raft_consensus.cc:729] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00dd182ab40843dd9e5f210aea6300d9, State: Initialized, Role: FOLLOWER
I20260812 00:15:41.231220 31146 consensus_queue.cc:260] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.231814 31146 ts_tablet_manager.cc:1428] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:41.231935 31119 heartbeater.cc:503] Master 127.31.227.62:34101 was elected leader, sending a full tablet report...
W20260812 00:15:41.307636 30916 tablet.cc:2369] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:41.431702 30684 tablet.cc:2369] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:15:41.435678 31148 raft_consensus.cc:482] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:41.435822 31148 raft_consensus.cc:504] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.436213 31148 leader_election.cc:288] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00dd182ab40843dd9e5f210aea6300d9 (127.31.227.3:43853), c990dba04a0244a7a5b43de9d63a7408 (127.31.227.1:41917)
W20260812 00:15:41.439215 31121 tablet.cc:2369] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:15:41.439285 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.436436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:43853 (local address 127.31.227.2:38991)
0812 00:15:41.436528 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.436541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.436590 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.436875 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.436875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.436876 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:41.436936 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.436938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.437623 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.437625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.438382 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.438384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.439008 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.439009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.439079 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.439082 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.439152 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.439192 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":27}
I20260812 00:15:41.439332 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.436669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:38991 (local address 127.31.227.3:43853)
0812 00:15:41.436730 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.436732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.436747 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:15:41.436796 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.436797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.436807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.436862 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:41.437013 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.437494 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.438499 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.438909 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.439006 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.439014 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.439159 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.439209 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:15:41.439404 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.436542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:46585 (local address 127.31.227.1:41917)
0812 00:15:41.436592 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.436594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.436784 (+ 190us) server_negotiation.cc:407] Connection header received
0812 00:15:41.436829 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.436829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.436841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.436896 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:41.437097 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.437626 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.438586 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.439005 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.439086 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.439095 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.439280 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.439319 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:15:41.439639 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.436446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41917 (local address 127.31.227.2:46585)
0812 00:15:41.436713 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.436723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:41.436776 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:41.436937 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:41.436939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:41.436940 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:41.437016 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.437018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.437712 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.437713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.438494 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:41.438496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.439106 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.439107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:41.439185 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.439189 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:41.439517 (+ 328us) client_negotiation.cc:240] Negotiation successful
0812 00:15:41.439569 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":12,"thread_start_us":138,"threads_started":1}
I20260812 00:15:41.439656 31044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d2771c61a3474ca99bfd3465c16463" candidate_uuid: "b8c2d536dbb34ad69424628110844a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00dd182ab40843dd9e5f210aea6300d9" is_pre_election: true
I20260812 00:15:41.439805 31044 raft_consensus.cc:2454] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c2d536dbb34ad69424628110844a1e in term 0.
I20260812 00:15:41.440097 30714 leader_election.cc:302] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00dd182ab40843dd9e5f210aea6300d9, b8c2d536dbb34ad69424628110844a1e; no voters:
I20260812 00:15:41.440088 30597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d2771c61a3474ca99bfd3465c16463" candidate_uuid: "b8c2d536dbb34ad69424628110844a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c990dba04a0244a7a5b43de9d63a7408" is_pre_election: true
I20260812 00:15:41.440189 30597 raft_consensus.cc:2454] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c2d536dbb34ad69424628110844a1e in term 0.
I20260812 00:15:41.440239 31148 raft_consensus.cc:2781] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:41.440311 31148 raft_consensus.cc:482] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:41.440335 31148 raft_consensus.cc:3037] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:41.441288 31148 raft_consensus.cc:504] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.441412 31148 leader_election.cc:288] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [CANDIDATE]: Term 1 election: Requested vote from peers 00dd182ab40843dd9e5f210aea6300d9 (127.31.227.3:43853), c990dba04a0244a7a5b43de9d63a7408 (127.31.227.1:41917)
I20260812 00:15:41.441560 31044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d2771c61a3474ca99bfd3465c16463" candidate_uuid: "b8c2d536dbb34ad69424628110844a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00dd182ab40843dd9e5f210aea6300d9"
I20260812 00:15:41.441581 30597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d2771c61a3474ca99bfd3465c16463" candidate_uuid: "b8c2d536dbb34ad69424628110844a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c990dba04a0244a7a5b43de9d63a7408"
I20260812 00:15:41.441635 31044 raft_consensus.cc:3037] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:41.441646 30597 raft_consensus.cc:3037] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:41.442708 30597 raft_consensus.cc:2454] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c2d536dbb34ad69424628110844a1e in term 1.
I20260812 00:15:41.442708 31044 raft_consensus.cc:2454] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c2d536dbb34ad69424628110844a1e in term 1.
I20260812 00:15:41.442929 30712 leader_election.cc:302] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8c2d536dbb34ad69424628110844a1e, c990dba04a0244a7a5b43de9d63a7408; no voters:
I20260812 00:15:41.443040 31148 raft_consensus.cc:2781] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:41.443215 31148 raft_consensus.cc:686] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 1 LEADER]: Becoming Leader. State: Replica: b8c2d536dbb34ad69424628110844a1e, State: Running, Role: LEADER
I20260812 00:15:41.443311 31148 consensus_queue.cc:237] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } }
I20260812 00:15:41.444132 30345 catalog_manager.cc:5495] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8c2d536dbb34ad69424628110844a1e (127.31.227.2). New cstate: current_term: 1 leader_uuid: "b8c2d536dbb34ad69424628110844a1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8c2d536dbb34ad69424628110844a1e" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 41445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 } health_report { overall_health: UNKNOWN } } }
I20260812 00:15:41.501726 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.499192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.31.227.2:41445)
0812 00:15:41.499262 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:41.499264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:41.499274 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:41.499322 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:41.499323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:41.499347 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:41.499426 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:41.499577 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.500074 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.500847 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:41.501060 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:41.501136 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:41.501309 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:41.501425 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:41.501511 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:41.501597 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 00:15:41.501616 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:15:41.503639 30597 raft_consensus.cc:1261] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 1 FOLLOWER]: Refusing update from remote peer b8c2d536dbb34ad69424628110844a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:41.503675 31044 raft_consensus.cc:1261] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 1 FOLLOWER]: Refusing update from remote peer b8c2d536dbb34ad69424628110844a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:41.504058 31152 consensus_queue.cc:1035] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c990dba04a0244a7a5b43de9d63a7408" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:41.504231 31148 consensus_queue.cc:1035] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00dd182ab40843dd9e5f210aea6300d9" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:41.506865 31165 mvcc.cc:205] Tried to move back new op lower bound from 7317478365196177408 to 7317478364952309760. Current Snapshot: MvccSnapshot[applied={T|T < 7317478365196177408}]
I20260812 00:15:41.508404 31167 mvcc.cc:205] Tried to move back new op lower bound from 7317478365196177408 to 7317478364952309760. Current Snapshot: MvccSnapshot[applied={T|T < 7317478365196177408 or (T in {7317478365196177408})}]
I20260812 00:15:41.508617 30344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:41.508710 30344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:41.509483 30344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 378f55bc30944cafb9be6650dff7d8fd: Sending DeleteTablet for 3 replicas of tablet 42d2771c61a3474ca99bfd3465c16463
I20260812 00:15:41.509819 30796 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d2771c61a3474ca99bfd3465c16463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:41 UTC) from {username='slave'} at 127.0.0.1:38496
I20260812 00:15:41.509841 31011 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d2771c61a3474ca99bfd3465c16463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:41 UTC) from {username='slave'} at 127.0.0.1:39618
I20260812 00:15:41.510095 31179 tablet_replica.cc:330] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: stopping tablet replica
I20260812 00:15:41.510084 30562 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d2771c61a3474ca99bfd3465c16463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:41 UTC) from {username='slave'} at 127.0.0.1:46556
I20260812 00:15:41.510203 31179 raft_consensus.cc:2229] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:41.510317 31181 tablet_replica.cc:330] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408: stopping tablet replica
I20260812 00:15:41.510351 31179 raft_consensus.cc:2258] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:41.510496 31181 raft_consensus.cc:2229] T 42d2771c61a3474ca99bfd3465c16463 P c990dba04a0244a7a5b43de9d63a7408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:41.510551 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 30452
I20260812 00:15:41.510807 31179 ts_tablet_manager.cc:1905] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:41.514791 31179 ts_tablet_manager.cc:1918] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:41.514873 31179 log.cc:1196] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/42d2771c61a3474ca99bfd3465c16463
I20260812 00:15:41.515084 31179 ts_tablet_manager.cc:1939] T 42d2771c61a3474ca99bfd3465c16463 P 00dd182ab40843dd9e5f210aea6300d9: Deleting consensus metadata
I20260812 00:15:41.517982 30324 catalog_manager.cc:4841] TS 00dd182ab40843dd9e5f210aea6300d9 (127.31.227.3:43853): tablet 42d2771c61a3474ca99bfd3465c16463 (table test-table [id=2bdadbcbfddd47e1b95a35677c39e6f1]) successfully deleted
I20260812 00:15:41.522491 31178 tablet_replica.cc:330] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: stopping tablet replica
I20260812 00:15:41.522629 31178 raft_consensus.cc:2229] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:15:41.522745 31178 raft_consensus.cc:2258] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:41.523260 31178 ts_tablet_manager.cc:1905] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:41.524667 31178 ts_tablet_manager.cc:1918] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:41.524761 31178 log.cc:1196] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/42d2771c61a3474ca99bfd3465c16463
I20260812 00:15:41.525036 31178 ts_tablet_manager.cc:1939] T 42d2771c61a3474ca99bfd3465c16463 P b8c2d536dbb34ad69424628110844a1e: Deleting consensus metadata
I20260812 00:15:41.525537 30325 catalog_manager.cc:4841] TS b8c2d536dbb34ad69424628110844a1e (127.31.227.2:41445): tablet 42d2771c61a3474ca99bfd3465c16463 (table test-table [id=2bdadbcbfddd47e1b95a35677c39e6f1]) successfully deleted
W20260812 00:15:41.528066 30323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:41917 (error 108)
I20260812 00:15:41.528329 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 30688
I20260812 00:15:41.530558 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:41.528460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41917 (local address 127.0.0.1:46562)
0812 00:15:41.530471 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:41.530521 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1984}
W20260812 00:15:41.534086 30323 catalog_manager.cc:4584] TS c990dba04a0244a7a5b43de9d63a7408 (127.31.227.1:41917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42d2771c61a3474ca99bfd3465c16463: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41917: connect: Connection refused (error 111)
I20260812 00:15:41.556713 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 30919
I20260812 00:15:41.565884 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 30302
2026-08-12T00:15:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:41 dist-test-slave-zr1t krb5kdc[31186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:41 dist-test-slave-zr1t krb5kdc[31186](info): set up 2 sockets
Aug 12 00:15:41 dist-test-slave-zr1t krb5kdc[31186](info): commencing operation
krb5kdc: starting...
W20260812 00:15:42.823839 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:42 dist-test-slave-zr1t krb5kdc[31186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:15:42.944519 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42239
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:43.028668 31208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:43.028874 31208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:43.028898 31208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:43.030089 31208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:43.030136 31208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:43.030151 31208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:43.030164 31208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:43.030176 31208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:43.031803 31208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:43.032078 31208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:43.032323 31208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:43.037282 31220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.037267 31217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.037438 31208 server_base.cc:1034] running on GCE node
W20260812 00:15:43.037272 31222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.037791 31208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:43.038107 31208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:43.039304 31208 hybrid_clock.cc:648] HybridClock initialized: now 1786493743039280 us; error 42 us; skew 500 ppm
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:43.046869 31208 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:15:43.047475 31208 webserver.cc:460] Webserver started at http://127.31.227.62:36057/ using document root <none> and password file <none>
I20260812 00:15:43.047763 31208 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:43.047832 31208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:43.047983 31208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:43.049579 31208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "4152b1fde0594a839c1e65a25b972219"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "518b2bacecb2da99c8bdee5642da512c"
server_key_iv: "c42b26d7fab7d714ae085fba3332bd25"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.050030 31208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "4152b1fde0594a839c1e65a25b972219"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "518b2bacecb2da99c8bdee5642da512c"
server_key_iv: "c42b26d7fab7d714ae085fba3332bd25"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.051851 31208 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260812 00:15:43.056565 31234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.056790 31208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 00:15:43.056893 31208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "4152b1fde0594a839c1e65a25b972219"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "518b2bacecb2da99c8bdee5642da512c"
server_key_iv: "c42b26d7fab7d714ae085fba3332bd25"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.056988 31208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:43.090204 31208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:43.092197 31208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:43.092358 31208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:43.096397 31208 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42239
I20260812 00:15:43.096437 31323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42239 every 8 connection(s)
I20260812 00:15:43.096822 31208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:43.097805 31325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:43.099817 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 31208
I20260812 00:15:43.099964 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:43.100174 32652 external_mini_cluster.cc:1428] Setting key 7ba10186c698f0b3e297c47c68f07b06
I20260812 00:15:43.100204 31325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: Bootstrap starting.
I20260812 00:15:43.101037 31325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:43.101430 31325 log.cc:824] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:43.102239 31325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: No bootstrap required, opened a new log
I20260812 00:15:43.104586 31325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152b1fde0594a839c1e65a25b972219" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42239 } }
I20260812 00:15:43.104745 31325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:43.104792 31325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4152b1fde0594a839c1e65a25b972219, State: Initialized, Role: FOLLOWER
I20260812 00:15:43.104964 31325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152b1fde0594a839c1e65a25b972219" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42239 } }
I20260812 00:15:43.105054 31325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:43.105084 31325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:43.105121 31325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:43.105850 31325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152b1fde0594a839c1e65a25b972219" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42239 } }
I20260812 00:15:43.105978 31325 leader_election.cc:302] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4152b1fde0594a839c1e65a25b972219; no voters:
I20260812 00:15:43.106144 31325 leader_election.cc:288] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:43.106230 31331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:43.106393 31331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [term 1 LEADER]: Becoming Leader. State: Replica: 4152b1fde0594a839c1e65a25b972219, State: Running, Role: LEADER
I20260812 00:15:43.106411 31325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:43.106494 31331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152b1fde0594a839c1e65a25b972219" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42239 } }
I20260812 00:15:43.106856 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4152b1fde0594a839c1e65a25b972219. Latest consensus state: current_term: 1 leader_uuid: "4152b1fde0594a839c1e65a25b972219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152b1fde0594a839c1e65a25b972219" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42239 } } }
I20260812 00:15:43.106943 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:43.107115 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4152b1fde0594a839c1e65a25b972219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152b1fde0594a839c1e65a25b972219" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42239 } } }
I20260812 00:15:43.107209 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219 [sys.catalog]: This master's current role is: LEADER
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
W20260812 00:15:43.108747 31356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:43.108806 31356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:43.108870 31340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:43.109042 31340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:43.110654 31340 catalog_manager.cc:1347] Generated new cluster ID: df5eec274ab84bfdacd458c9ac6233f6
I20260812 00:15:43.110710 31340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:43.112509 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.100769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.31.227.62:42239)
0812 00:15:43.100974 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.100984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.101034 (+ 50us) server_negotiation.cc:407] Connection header received
0812 00:15:43.101756 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.101765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.101815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.101975 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:43.102676 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.104094 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.104962 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.105610 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.105726 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.109055 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:43.109059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:43.109059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:43.109071 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:43.110971 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.111331 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.111334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.111335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.111422 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.111632 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.111633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.111633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.111867 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:43.112034 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.112154 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.112295 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260812 00:15:43.127844 31340 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:43.128566 31340 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:43.134426 31340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: Generated new TSK 0
I20260812 00:15:43.134621 31340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:15:43.160251 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42239
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:43.245973 31362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:43.246169 31362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:43.246193 31362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:43.247315 31362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:43.247356 31362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:43.247408 31362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:43.248914 31362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:43.249156 31362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:43.249387 31362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:43.254410 31378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.254464 31374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.254390 31376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.254549 31362 server_base.cc:1034] running on GCE node
I20260812 00:15:43.254845 31362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:43.255098 31362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:43.256287 31362 hybrid_clock.cc:648] HybridClock initialized: now 1786493743256273 us; error 42 us; skew 500 ppm
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:43.258875 31362 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:15:43.259290 31362 webserver.cc:460] Webserver started at http://127.31.227.1:36085/ using document root <none> and password file <none>
I20260812 00:15:43.259516 31362 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:43.259582 31362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:43.259732 31362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:43.260759 31362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "72cea10b7f4e461aa4d15e6ce6e351ad"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "181fa82a8bfa5ae2bdea826ac6b5265f"
server_key_iv: "50be927b631d9892c2a012980fad6a5e"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.261134 31362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "72cea10b7f4e461aa4d15e6ce6e351ad"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "181fa82a8bfa5ae2bdea826ac6b5265f"
server_key_iv: "50be927b631d9892c2a012980fad6a5e"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.262630 31362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:43.263362 31389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.263540 31362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:15:43.263648 31362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "72cea10b7f4e461aa4d15e6ce6e351ad"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "181fa82a8bfa5ae2bdea826ac6b5265f"
server_key_iv: "50be927b631d9892c2a012980fad6a5e"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.263720 31362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:43.275238 31362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:43.277125 31362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:43.277294 31362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:43.277583 31362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:43.277915 31362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:43.277969 31362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.278012 31362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:43.278040 31362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.283896 31362 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:41699
I20260812 00:15:43.283934 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:41699 every 8 connection(s)
I20260812 00:15:43.284261 31362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:15:43.285773 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 31362
I20260812 00:15:43.285903 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:15:43.286113 32652 external_mini_cluster.cc:1428] Setting key 32358200a1d070c897c0a840ec9f0c75
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:43.294430 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.285017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:52469 (local address 127.31.227.62:42239)
0812 00:15:43.285083 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.285085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.285316 (+ 231us) server_negotiation.cc:407] Connection header received
0812 00:15:43.285956 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.285958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.285971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.286047 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:43.287237 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.287749 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.288749 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.288964 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.289048 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.291262 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:43.291264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:43.291264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:43.291272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:43.292984 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.293372 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.293373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.293374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.293440 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.293738 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.293739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.293739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.293894 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:43.293978 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.294227 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.294322 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:15:43.294445 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.284987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42239 (local address 127.31.227.1:52469)
0812 00:15:43.285176 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.285216 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.285813 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.286126 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.286137 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.286936 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:43.287140 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.287146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.287847 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.287849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.288648 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.288649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.289063 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.289064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.289185 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.289732 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:43.289741 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:43.291173 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:43.293078 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:43.293085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:43.293087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:43.293297 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:43.293551 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:43.293552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:43.293552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:43.293666 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:43.293976 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:43.293981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:43.294088 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.294160 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.294273 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":118,"threads_started":1}
I20260812 00:15:43.294987 31584 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42239
I20260812 00:15:43.295094 31584 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:43.295291 31584 heartbeater.cc:511] Master 127.31.227.62:42239 requested a full tablet report, sending...
I20260812 00:15:43.295809 31265 ts_manager.cc:194] Registered new tserver with Master: 72cea10b7f4e461aa4d15e6ce6e351ad (127.31.227.1:41699)
I20260812 00:15:43.296411 31265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:52469
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:15:43.312167 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42239
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:43.394527 31595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:43.394769 31595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:43.394809 31595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:43.395973 31595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:43.396042 31595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:43.396087 31595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:43.397593 31595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:43.397866 31595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:43.398102 31595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:43.403400 31602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.403450 31604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.403393 31606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.403569 31595 server_base.cc:1034] running on GCE node
I20260812 00:15:43.403812 31595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:43.404079 31595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:43.405256 31595 hybrid_clock.cc:648] HybridClock initialized: now 1786493743405242 us; error 25 us; skew 500 ppm
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:43.408166 31595 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:15:43.408746 31595 webserver.cc:460] Webserver started at http://127.31.227.2:42807/ using document root <none> and password file <none>
I20260812 00:15:43.408993 31595 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:43.409039 31595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:43.409133 31595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:43.410223 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "694c2b76a9294d16974167e1fab89aa3"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "6eebca7e7424f9c30604b5a5ab4acbc0"
server_key_iv: "bf6aced2fd24ff228de959f838346a88"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.410538 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "694c2b76a9294d16974167e1fab89aa3"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "6eebca7e7424f9c30604b5a5ab4acbc0"
server_key_iv: "bf6aced2fd24ff228de959f838346a88"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.411818 31595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:43.412487 31616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.412658 31595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:43.412765 31595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "694c2b76a9294d16974167e1fab89aa3"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "6eebca7e7424f9c30604b5a5ab4acbc0"
server_key_iv: "bf6aced2fd24ff228de959f838346a88"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.412843 31595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:43.427868 31595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:43.429865 31595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:43.430039 31595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:43.430280 31595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:43.430606 31595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:43.430661 31595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.430697 31595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:43.430724 31595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.437114 31595 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:38751
I20260812 00:15:43.437158 31817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:38751 every 8 connection(s)
I20260812 00:15:43.437474 31595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:43.446187 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.438256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:58693 (local address 127.31.227.62:42239)
0812 00:15:43.438307 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.438309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.438507 (+ 198us) server_negotiation.cc:407] Connection header received
0812 00:15:43.439000 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.439002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.439012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.439064 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:43.439989 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.440479 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.441494 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.441731 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.441784 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.443831 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:43.443833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:43.443833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:43.443840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:43.445009 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.445383 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.445384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.445384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.445432 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.445664 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.445665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.445665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.445767 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:43.445823 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.446061 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.446122 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:15:43.446401 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.438221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42239 (local address 127.31.227.2:58693)
0812 00:15:43.438399 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.438436 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.438914 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.439120 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.439123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.439703 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:43.439913 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.439917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.440564 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.440565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.441419 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.441422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.441807 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.441808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.441912 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.442405 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:43.442414 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:43.443749 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:43.445079 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:43.445085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:43.445087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:43.445299 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:43.445491 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:43.445491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:43.445492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:43.445607 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:43.445826 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:43.445831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:43.445943 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.446030 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.446145 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260812 00:15:43.446923 31818 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42239
I20260812 00:15:43.447000 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 31595
I20260812 00:15:43.447031 31818 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:43.447119 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:15:43.447227 31818 heartbeater.cc:511] Master 127.31.227.62:42239 requested a full tablet report, sending...
I20260812 00:15:43.447345 32652 external_mini_cluster.cc:1428] Setting key 44c1e0545e0ed3e92c2e9f8f8160e1ea
I20260812 00:15:43.447670 31265 ts_manager.cc:194] Registered new tserver with Master: 694c2b76a9294d16974167e1fab89aa3 (127.31.227.2:38751)
I20260812 00:15:43.448221 31265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:58693
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:15:43.473776 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42239
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:43.558218 31824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:43.558403 31824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:43.558423 31824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:43.559499 31824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:43.559540 31824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:43.559564 31824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:43.560899 31824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:42239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:43.561076 31824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:43.561257 31824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:43.566201 31838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.566237 31830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.566237 31824 server_base.cc:1034] running on GCE node
W20260812 00:15:43.566509 31831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.566778 31824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:43.567073 31824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:43.568220 31824 hybrid_clock.cc:648] HybridClock initialized: now 1786493743568205 us; error 29 us; skew 500 ppm
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:43.570604 31824 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:15:43.571072 31824 webserver.cc:460] Webserver started at http://127.31.227.3:32775/ using document root <none> and password file <none>
I20260812 00:15:43.571280 31824 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:43.571343 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:43.571485 31824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:43.572486 31824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "247d3a6e20a845d09056fc9b693ecd74"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "258f51962dc5ffac67c753663a62451d"
server_key_iv: "65d7004a999c1d62f1fed38868e9397f"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.572847 31824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "247d3a6e20a845d09056fc9b693ecd74"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "258f51962dc5ffac67c753663a62451d"
server_key_iv: "65d7004a999c1d62f1fed38868e9397f"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.574214 31824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:43.574900 31851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.575066 31824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.575166 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "247d3a6e20a845d09056fc9b693ecd74"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "258f51962dc5ffac67c753663a62451d"
server_key_iv: "65d7004a999c1d62f1fed38868e9397f"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.575237 31824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:43.586978 31824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:43.588590 31824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:43.588735 31824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:43.588967 31824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:43.589258 31824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:43.589308 31824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.589339 31824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:43.589377 31824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.595463 31824 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41173
I20260812 00:15:43.595512 32038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41173 every 8 connection(s)
I20260812 00:15:43.595782 31824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:15:43.599140 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 31824
I20260812 00:15:43.599257 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:15:43.599462 32652 external_mini_cluster.cc:1428] Setting key 0fa57bbc07efd5864ded794c10486f37
Aug 12 00:15:43 dist-test-slave-zr1t krb5kdc[31186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:43.604729 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.596398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:48409 (local address 127.31.227.62:42239)
0812 00:15:43.596475 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.596478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.596675 (+ 197us) server_negotiation.cc:407] Connection header received
0812 00:15:43.597121 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.597122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.597133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.597183 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:43.598132 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.598656 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.600001 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.600303 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.600371 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.602451 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:43.602453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:43.602453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:43.602459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:43.603626 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.603980 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.603980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.603981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.604029 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.604236 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.604237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.604237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.604334 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:43.604384 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.604608 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.604668 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:15:43.604894 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.596371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:42239 (local address 127.31.227.3:48409)
0812 00:15:43.596563 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.596596 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.597016 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.597231 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.597234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.597864 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:43.598056 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.598060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.598735 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.598736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.599912 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.599922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.600391 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.600392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.600547 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.600982 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:43.600989 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:43.602364 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:43.603699 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:43.603705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:43.603706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:43.603915 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:43.604088 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:43.604089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:43.604089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:43.604186 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:43.604388 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:43.604393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:43.604505 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.604583 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.604696 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
I20260812 00:15:43.605332 32042 heartbeater.cc:347] Connected to a master server at 127.31.227.62:42239
I20260812 00:15:43.605440 32042 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:43.605640 32042 heartbeater.cc:511] Master 127.31.227.62:42239 requested a full tablet report, sending...
I20260812 00:15:43.605995 31265 ts_manager.cc:194] Registered new tserver with Master: 247d3a6e20a845d09056fc9b693ecd74 (127.31.227.3:41173)
I20260812 00:15:43.606338 31265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:48409
I20260812 00:15:43.611258 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:43.618314 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.612702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.31.227.62:42239)
0812 00:15:43.612760 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.612762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.612822 (+ 60us) server_negotiation.cc:407] Connection header received
0812 00:15:43.612861 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.612862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.612870 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.612934 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:43.613425 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.613947 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.614651 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.614823 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.614872 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.616160 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:43.616161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:43.616161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:43.616168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:43.617359 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.617735 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.617736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.617736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.617784 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:43.617911 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:43.617911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:43.617911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:43.618018 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:43.618092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.618203 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.618254 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:15:43.619750 31265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:43.621258 31265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:43.627785 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.623527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41699 (local address 127.0.0.1:53914)
0812 00:15:43.623789 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.623801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.623850 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.624295 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.624298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.624300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:43.624373 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.624376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.625370 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.625371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.626658 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.626663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.627463 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.627464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.627550 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.627564 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.627660 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.627715 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":41,"threads_started":1}
I20260812 00:15:43.627823 32066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.623930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.31.227.1:41699)
0812 00:15:43.624117 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.624122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.624131 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:43.624165 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.624166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.624179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.624272 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:43.624508 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.625246 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.626760 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.627370 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.627459 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.627520 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.627681 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.627722 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260812 00:15:43.628180 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.623481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:38751 (local address 127.0.0.1:40002)
0812 00:15:43.623732 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.623773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.623851 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.624605 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.624606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.624607 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:43.624640 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.624641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.625954 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.625956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.626947 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.626950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.627834 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.627971 (+ 137us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.628028 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.628031 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.628061 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.628090 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":55,"threads_started":1}
I20260812 00:15:43.628408 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.623594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.31.227.2:38751)
0812 00:15:43.624433 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.624438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.624447 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:43.624488 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.624490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.624506 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.624712 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:43.624783 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.625850 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.627021 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.627736 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.628164 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.628237 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.628267 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.628331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":109,"threads_started":1}
I20260812 00:15:43.628475 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.623726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.31.227.3:41173)
0812 00:15:43.624123 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.624127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.624135 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:15:43.624168 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.624170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.624181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.624296 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:43.624591 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.625693 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.626959 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.627669 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.627743 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.627817 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.628348 (+ 531us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.628389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":91,"threads_started":1}
I20260812 00:15:43.628123 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.623488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41173 (local address 127.0.0.1:52924)
0812 00:15:43.623752 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.623787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.623849 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.624477 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.624478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.624480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:43.624512 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.624513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.625801 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.625802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.626871 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.626872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.627848 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.627849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.627941 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.627946 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.627996 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.628038 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":30,"thread_start_us":71,"threads_started":1}
I20260812 00:15:43.628609 31469 tablet_service.cc:1467] Processing CreateTablet for tablet dc6afea0681a4469891d253b565907ee (DEFAULT_TABLE table=test-table [id=4b7e841380da4e21bfaf0044b8968842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:43.628981 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6afea0681a4469891d253b565907ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:43.629012 31706 tablet_service.cc:1467] Processing CreateTablet for tablet dc6afea0681a4469891d253b565907ee (DEFAULT_TABLE table=test-table [id=4b7e841380da4e21bfaf0044b8968842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:43.629359 31706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6afea0681a4469891d253b565907ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:43.631835 32069 tablet_bootstrap.cc:492] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Bootstrap starting.
I20260812 00:15:43.632385 31924 tablet_service.cc:1467] Processing CreateTablet for tablet dc6afea0681a4469891d253b565907ee (DEFAULT_TABLE table=test-table [id=4b7e841380da4e21bfaf0044b8968842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:43.632525 32069 tablet_bootstrap.cc:654] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:43.632728 31924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6afea0681a4469891d253b565907ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:43.633011 32069 log.cc:824] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:43.633358 32068 tablet_bootstrap.cc:492] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: Bootstrap starting.
I20260812 00:15:43.633852 32069 tablet_bootstrap.cc:492] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: No bootstrap required, opened a new log
I20260812 00:15:43.633934 32069 ts_tablet_manager.cc:1397] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:43.634069 32068 tablet_bootstrap.cc:654] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:43.634581 32068 log.cc:824] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:43.635409 32068 tablet_bootstrap.cc:492] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: No bootstrap required, opened a new log
I20260812 00:15:43.635492 32068 ts_tablet_manager.cc:1397] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:43.636134 32069 raft_consensus.cc:348] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.636276 32069 raft_consensus.cc:374] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:43.636305 32069 raft_consensus.cc:729] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694c2b76a9294d16974167e1fab89aa3, State: Initialized, Role: FOLLOWER
I20260812 00:15:43.636229 32072 tablet_bootstrap.cc:492] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Bootstrap starting.
I20260812 00:15:43.636423 32069 consensus_queue.cc:260] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.636720 31818 heartbeater.cc:503] Master 127.31.227.62:42239 was elected leader, sending a full tablet report...
I20260812 00:15:43.636718 32069 ts_tablet_manager.cc:1428] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:15:43.637233 32072 tablet_bootstrap.cc:654] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:43.637918 32072 log.cc:824] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:43.637809 32068 raft_consensus.cc:348] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.638026 32068 raft_consensus.cc:374] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:43.638075 32068 raft_consensus.cc:729] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cea10b7f4e461aa4d15e6ce6e351ad, State: Initialized, Role: FOLLOWER
I20260812 00:15:43.638202 32068 consensus_queue.cc:260] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.638474 32068 ts_tablet_manager.cc:1428] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:43.638595 31584 heartbeater.cc:503] Master 127.31.227.62:42239 was elected leader, sending a full tablet report...
I20260812 00:15:43.638695 32072 tablet_bootstrap.cc:492] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: No bootstrap required, opened a new log
I20260812 00:15:43.638763 32072 ts_tablet_manager.cc:1397] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:15:43.640074 32072 raft_consensus.cc:348] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.640218 32072 raft_consensus.cc:374] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:43.640260 32072 raft_consensus.cc:729] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d3a6e20a845d09056fc9b693ecd74, State: Initialized, Role: FOLLOWER
I20260812 00:15:43.640375 32072 consensus_queue.cc:260] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.640648 32072 ts_tablet_manager.cc:1428] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:43.640678 32042 heartbeater.cc:503] Master 127.31.227.62:42239 was elected leader, sending a full tablet report...
W20260812 00:15:43.688102 31819 tablet.cc:2369] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:15:43.745803 32076 raft_consensus.cc:482] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:43.745940 32076 raft_consensus.cc:504] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.746308 32076 leader_election.cc:288] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694c2b76a9294d16974167e1fab89aa3 (127.31.227.2:38751), 247d3a6e20a845d09056fc9b693ecd74 (127.31.227.3:41173)
I20260812 00:15:43.749565 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.746589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:56181 (local address 127.31.227.3:41173)
0812 00:15:43.746665 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.746667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.746679 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:15:43.746752 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.746753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.746764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.746838 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:43.747070 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.747633 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.748647 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.749063 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.749122 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.749130 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.749423 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.749465 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:15:43.749578 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.746595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:42155 (local address 127.31.227.2:38751)
0812 00:15:43.746647 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.746649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.746933 (+ 284us) server_negotiation.cc:407] Connection header received
0812 00:15:43.746967 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.746969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.746979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.747048 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:43.747214 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.747698 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.748703 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.749085 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.749277 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.749284 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.749445 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.749489 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:15:43.749578 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.746511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41173 (local address 127.31.227.1:56181)
0812 00:15:43.746609 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.746624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.746682 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.746871 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.746872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.746874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:43.746986 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.746988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.747720 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.747721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.748527 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.748529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.749250 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.749252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.749336 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.749344 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.749388 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.749459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":31}
I20260812 00:15:43.749593 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.746521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:38751 (local address 127.31.227.1:42155)
0812 00:15:43.746853 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.746863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:43.746922 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:43.747068 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:43.747069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:43.747071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:43.747143 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.747144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.747783 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.747784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.748575 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:43.748577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.749287 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.749288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:43.749344 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.749347 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:43.749436 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 00:15:43.749471 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":145,"thread_start_us":110,"threads_started":1}
I20260812 00:15:43.750103 31959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6afea0681a4469891d253b565907ee" candidate_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d3a6e20a845d09056fc9b693ecd74" is_pre_election: true
I20260812 00:15:43.750278 31959 raft_consensus.cc:2454] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72cea10b7f4e461aa4d15e6ce6e351ad in term 0.
I20260812 00:15:43.750264 31736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6afea0681a4469891d253b565907ee" candidate_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694c2b76a9294d16974167e1fab89aa3" is_pre_election: true
I20260812 00:15:43.750406 31736 raft_consensus.cc:2454] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72cea10b7f4e461aa4d15e6ce6e351ad in term 0.
I20260812 00:15:43.750536 31393 leader_election.cc:302] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d3a6e20a845d09056fc9b693ecd74, 72cea10b7f4e461aa4d15e6ce6e351ad; no voters:
I20260812 00:15:43.750715 32076 raft_consensus.cc:2781] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:43.750763 32076 raft_consensus.cc:482] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:43.750780 32076 raft_consensus.cc:3037] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:43.751525 32076 raft_consensus.cc:504] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.751672 32076 leader_election.cc:288] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [CANDIDATE]: Term 1 election: Requested vote from peers 694c2b76a9294d16974167e1fab89aa3 (127.31.227.2:38751), 247d3a6e20a845d09056fc9b693ecd74 (127.31.227.3:41173)
I20260812 00:15:43.751856 31736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6afea0681a4469891d253b565907ee" candidate_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694c2b76a9294d16974167e1fab89aa3"
I20260812 00:15:43.751856 31959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6afea0681a4469891d253b565907ee" candidate_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d3a6e20a845d09056fc9b693ecd74"
I20260812 00:15:43.751935 31959 raft_consensus.cc:3037] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:43.751937 31736 raft_consensus.cc:3037] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:43.752861 31959 raft_consensus.cc:2454] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72cea10b7f4e461aa4d15e6ce6e351ad in term 1.
I20260812 00:15:43.752861 31736 raft_consensus.cc:2454] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72cea10b7f4e461aa4d15e6ce6e351ad in term 1.
I20260812 00:15:43.753055 31391 leader_election.cc:302] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694c2b76a9294d16974167e1fab89aa3, 72cea10b7f4e461aa4d15e6ce6e351ad; no voters:
I20260812 00:15:43.753161 32076 raft_consensus.cc:2781] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:43.753331 32076 raft_consensus.cc:686] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 1 LEADER]: Becoming Leader. State: Replica: 72cea10b7f4e461aa4d15e6ce6e351ad, State: Running, Role: LEADER
I20260812 00:15:43.753435 32076 consensus_queue.cc:237] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } }
I20260812 00:15:43.754369 31265 catalog_manager.cc:5495] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 72cea10b7f4e461aa4d15e6ce6e351ad (127.31.227.1). New cstate: current_term: 1 leader_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72cea10b7f4e461aa4d15e6ce6e351ad" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 41699 } health_report { overall_health: HEALTHY } } }
I20260812 00:15:43.773190 32066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.770751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.31.227.1:41699)
0812 00:15:43.770815 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:43.770818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:43.770828 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:43.770914 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:43.770916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:43.770932 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:43.771010 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:43.771150 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.771637 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.772394 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:43.772594 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:43.772668 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:43.772806 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:43.772901 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:43.773009 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:43.773067 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 00:15:43.773078 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:15:43.775013 31736 raft_consensus.cc:1261] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 1 FOLLOWER]: Refusing update from remote peer 72cea10b7f4e461aa4d15e6ce6e351ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:43.775012 31959 raft_consensus.cc:1261] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 1 FOLLOWER]: Refusing update from remote peer 72cea10b7f4e461aa4d15e6ce6e351ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:43.775372 32076 consensus_queue.cc:1035] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "694c2b76a9294d16974167e1fab89aa3" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:43.775528 32079 consensus_queue.cc:1035] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "247d3a6e20a845d09056fc9b693ecd74" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:43.778795 32086 mvcc.cc:205] Tried to move back new op lower bound from 7317478374499905536 to 7317478374414848000. Current Snapshot: MvccSnapshot[applied={T|T < 7317478374499905536}]
I20260812 00:15:43.779244 32085 mvcc.cc:205] Tried to move back new op lower bound from 7317478374499905536 to 7317478374414848000. Current Snapshot: MvccSnapshot[applied={T|T < 7317478374499905536}]
I20260812 00:15:43.780388 31263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:43.780472 31263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:43.781203 31263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4152b1fde0594a839c1e65a25b972219: Sending DeleteTablet for 3 replicas of tablet dc6afea0681a4469891d253b565907ee
I20260812 00:15:43.781569 31924 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6afea0681a4469891d253b565907ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:43 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:52924
I20260812 00:15:43.781558 31706 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6afea0681a4469891d253b565907ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:43 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:40002
I20260812 00:15:43.781608 31469 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6afea0681a4469891d253b565907ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:43 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:53914
I20260812 00:15:43.781783 32097 tablet_replica.cc:330] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: stopping tablet replica
I20260812 00:15:43.781829 32100 tablet_replica.cc:330] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: stopping tablet replica
I20260812 00:15:43.781883 32097 raft_consensus.cc:2229] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:43.781905 32100 raft_consensus.cc:2229] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:15:43.781983 32100 raft_consensus.cc:2258] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:43.781994 32097 raft_consensus.cc:2258] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:43.782347 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 31362
I20260812 00:15:43.782549 32100 ts_tablet_manager.cc:1905] T dc6afea0681a4469891d253b565907ee P 72cea10b7f4e461aa4d15e6ce6e351ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:43.783267 32097 ts_tablet_manager.cc:1905] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:43.785439 32097 ts_tablet_manager.cc:1918] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:43.785579 32097 log.cc:1196] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/dc6afea0681a4469891d253b565907ee
I20260812 00:15:43.785926 32097 ts_tablet_manager.cc:1939] T dc6afea0681a4469891d253b565907ee P 247d3a6e20a845d09056fc9b693ecd74: Deleting consensus metadata
I20260812 00:15:43.786386 31240 catalog_manager.cc:4841] TS 247d3a6e20a845d09056fc9b693ecd74 (127.31.227.3:41173): tablet dc6afea0681a4469891d253b565907ee (table test-table [id=4b7e841380da4e21bfaf0044b8968842]) successfully deleted
I20260812 00:15:43.792375 32099 tablet_replica.cc:330] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: stopping tablet replica
I20260812 00:15:43.792546 32099 raft_consensus.cc:2229] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:43.792686 32099 raft_consensus.cc:2258] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:43.794428 32099 ts_tablet_manager.cc:1905] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:43.796204 32099 ts_tablet_manager.cc:1918] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:43.796335 32099 log.cc:1196] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/dc6afea0681a4469891d253b565907ee
I20260812 00:15:43.796653 32099 ts_tablet_manager.cc:1939] T dc6afea0681a4469891d253b565907ee P 694c2b76a9294d16974167e1fab89aa3: Deleting consensus metadata
I20260812 00:15:43.797125 31236 catalog_manager.cc:4841] TS 694c2b76a9294d16974167e1fab89aa3 (127.31.227.2:38751): tablet dc6afea0681a4469891d253b565907ee (table test-table [id=4b7e841380da4e21bfaf0044b8968842]) successfully deleted
W20260812 00:15:43.799760 31238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:41699 (error 108)
I20260812 00:15:43.800237 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:43.800124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:41699 (local address 127.0.0.1:53922)
0812 00:15:43.800175 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:43.800211 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 00:15:43.800369 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 31595
W20260812 00:15:43.800376 31238 catalog_manager.cc:4584] TS 72cea10b7f4e461aa4d15e6ce6e351ad (127.31.227.1:41699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc6afea0681a4469891d253b565907ee: Network error: Client connection negotiation failed: client connection to 127.31.227.1:41699: connect: Connection refused (error 111)
I20260812 00:15:43.810905 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 31824
I20260812 00:15:43.821693 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 31208
2026-08-12T00:15:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2255 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T00:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:43Z Disabled control of system clock
I20260812 00:15:43.854642 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45007
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:43.940281 32106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:43.940514 32106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:43.940537 32106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:43.941926 32106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:43.941977 32106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:43.941989 32106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:43.942036 32106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:43.942070 32106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:43.943565 32106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:45007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:43.943828 32106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:43.944131 32106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:43.949231 32113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:43.949246 32114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.949275 32106 server_base.cc:1034] running on GCE node
W20260812 00:15:43.949393 32117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:43.949697 32106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:43.950053 32106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:43.951208 32106 hybrid_clock.cc:648] HybridClock initialized: now 1786493743951205 us; error 38 us; skew 500 ppm
I20260812 00:15:43.952453 32106 webserver.cc:460] Webserver started at http://127.31.227.62:46717/ using document root <none> and password file <none>
I20260812 00:15:43.952701 32106 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:43.952747 32106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:43.952891 32106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:43.954051 32106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "d3a66105f09545e38a58740ec4db67b1"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "b0ef7e334548233c5088b0f806c6d697"
server_key_iv: "946a4cfbfdacf1d12b92103ccaa62f9f"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.954430 32106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "d3a66105f09545e38a58740ec4db67b1"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "b0ef7e334548233c5088b0f806c6d697"
server_key_iv: "946a4cfbfdacf1d12b92103ccaa62f9f"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.955706 32106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:43.956399 32125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:43.956585 32106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:43.956681 32106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "d3a66105f09545e38a58740ec4db67b1"
format_stamp: "Formatted at 2026-08-12 00:15:43 on dist-test-slave-zr1t"
server_key: "b0ef7e334548233c5088b0f806c6d697"
server_key_iv: "946a4cfbfdacf1d12b92103ccaa62f9f"
server_key_version: "encryptionkey@0"
I20260812 00:15:43.956760 32106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:43.995178 32106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:43.995635 32106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:43.995806 32106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:43.999980 32106 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:45007
I20260812 00:15:44.000018 32215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:45007 every 8 connection(s)
I20260812 00:15:44.000396 32106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:44.001400 32217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:44.003784 32217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1: Bootstrap starting.
I20260812 00:15:44.004385 32217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:44.004693 32217 log.cc:824] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:44.005391 32217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1: No bootstrap required, opened a new log
I20260812 00:15:44.007593 32217 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a66105f09545e38a58740ec4db67b1" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45007 } }
I20260812 00:15:44.007826 32217 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:44.007853 32217 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a66105f09545e38a58740ec4db67b1, State: Initialized, Role: FOLLOWER
I20260812 00:15:44.008059 32217 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a66105f09545e38a58740ec4db67b1" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45007 } }
I20260812 00:15:44.008129 32217 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:44.008186 32217 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:44.008236 32217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:44.008792 32217 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a66105f09545e38a58740ec4db67b1" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45007 } }
I20260812 00:15:44.008908 32217 leader_election.cc:302] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a66105f09545e38a58740ec4db67b1; no voters:
I20260812 00:15:44.009097 32217 leader_election.cc:288] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:44.009145 32220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:44.009352 32220 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [term 1 LEADER]: Becoming Leader. State: Replica: d3a66105f09545e38a58740ec4db67b1, State: Running, Role: LEADER
I20260812 00:15:44.009370 32217 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:44.009457 32220 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a66105f09545e38a58740ec4db67b1" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45007 } }
I20260812 00:15:44.009792 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 32106
I20260812 00:15:44.009919 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:44.010000 32226 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a66105f09545e38a58740ec4db67b1. Latest consensus state: current_term: 1 leader_uuid: "d3a66105f09545e38a58740ec4db67b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a66105f09545e38a58740ec4db67b1" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45007 } } }
I20260812 00:15:44.010090 32226 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:44.010197 32652 external_mini_cluster.cc:1428] Setting key 9ac554196f6209167aa29ad22cecfcbd
I20260812 00:15:44.010428 32229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:44.010622 32222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a66105f09545e38a58740ec4db67b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a66105f09545e38a58740ec4db67b1" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45007 } } }
I20260812 00:15:44.010706 32222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:44.011044 32229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:44.012718 32229 catalog_manager.cc:1347] Generated new cluster ID: 71a9f00f2cb94ce88c3f1ee65324f687
I20260812 00:15:44.012770 32229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:44.017843 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.010906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.31.227.62:45007)
0812 00:15:44.012354 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.012364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.012401 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:15:44.012522 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.012530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.012605 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.012754 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:44.013002 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.014391 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.015490 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.016144 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.016273 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.017024 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:44.017028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:44.017029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:44.017240 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:44.017282 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:44.017297 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:44.017404 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.017486 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.017607 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":95,"threads_started":1}
I20260812 00:15:44.037890 32229 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:44.038631 32229 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:44.044350 32229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1: Generated new TSK 0
I20260812 00:15:44.044538 32229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:44.053288 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45007
--builtin_ntp_servers=127.31.227.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:44.137430 32255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:44.137704 32255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:44.137748 32255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:44.138942 32255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:44.138983 32255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:44.139029 32255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:44.140368 32255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:44.140623 32255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:44.140951 32255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:44.146138 32266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:44.146152 32269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:44.146281 32265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:44.146147 32255 server_base.cc:1034] running on GCE node
I20260812 00:15:44.146562 32255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:44.146826 32255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:44.148016 32255 hybrid_clock.cc:648] HybridClock initialized: now 1786493744148000 us; error 26 us; skew 500 ppm
I20260812 00:15:44.149415 32255 webserver.cc:460] Webserver started at http://127.31.227.1:42057/ using document root <none> and password file <none>
I20260812 00:15:44.149710 32255 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:44.149784 32255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:44.149940 32255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:44.151037 32255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "144ec070138848e18e6924863f1fe56b"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "f8fb9822476407cd9298ded5ef35b9c6"
server_key_iv: "eb4a33a6d45c1944f52919b23cc6e3a2"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.151399 32255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "144ec070138848e18e6924863f1fe56b"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "f8fb9822476407cd9298ded5ef35b9c6"
server_key_iv: "eb4a33a6d45c1944f52919b23cc6e3a2"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.152710 32255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:15:44.153456 32279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.153638 32255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:44.153748 32255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "144ec070138848e18e6924863f1fe56b"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "f8fb9822476407cd9298ded5ef35b9c6"
server_key_iv: "eb4a33a6d45c1944f52919b23cc6e3a2"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.153822 32255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:44.161813 32255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:44.162091 32255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:44.162235 32255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:44.162468 32255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:44.162791 32255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:44.162844 32255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.162885 32255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:44.162914 32255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.168987 32255 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:38741
I20260812 00:15:44.169065 32473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:38741 every 8 connection(s)
I20260812 00:15:44.169360 32255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:15:44.174630 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.170097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:45009 (local address 127.31.227.62:45007)
0812 00:15:44.170144 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.170146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.170367 (+ 221us) server_negotiation.cc:407] Connection header received
0812 00:15:44.170917 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.170918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.170934 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.170998 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:44.171705 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.172224 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.173229 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.173441 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.173548 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.174333 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:44.174335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:44.174335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:44.174340 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:44.174353 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:44.174357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:44.174410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.174520 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.174550 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:15:44.174855 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.170060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45007 (local address 127.31.227.1:45009)
0812 00:15:44.170240 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.170273 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.170803 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.171063 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.171067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.171078 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:44.171603 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.171607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.172330 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.172331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.173130 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.173132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.173567 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.173568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.173706 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.174193 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:44.174193 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:44.174221 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:44.174243 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:44.174419 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:44.174424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:44.174428 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.174521 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.174629 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260812 00:15:44.175433 32474 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45007
I20260812 00:15:44.175518 32474 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:44.175719 32474 heartbeater.cc:511] Master 127.31.227.62:45007 requested a full tablet report, sending...
I20260812 00:15:44.176229 32149 ts_manager.cc:194] Registered new tserver with Master: 144ec070138848e18e6924863f1fe56b (127.31.227.1:38741)
I20260812 00:15:44.176841 32149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.1:45009
I20260812 00:15:44.179289 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 32255
I20260812 00:15:44.179416 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:15:44.179633 32652 external_mini_cluster.cc:1428] Setting key d2d1b2086d4e2de7b8b2f4ffc51f93ec
I20260812 00:15:44.180645 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45007
--builtin_ntp_servers=127.31.227.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:44.265580 32479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:44.265782 32479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:44.265805 32479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:44.266904 32479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:44.266940 32479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:44.266965 32479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:44.268429 32479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:44.268638 32479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:44.268887 32479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:44.274125 32488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:44.274130 32485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:44.274130 32492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:44.274372 32479 server_base.cc:1034] running on GCE node
I20260812 00:15:44.274600 32479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:44.274889 32479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:44.276084 32479 hybrid_clock.cc:648] HybridClock initialized: now 1786493744276079 us; error 39 us; skew 500 ppm
I20260812 00:15:44.277505 32479 webserver.cc:460] Webserver started at http://127.31.227.2:41841/ using document root <none> and password file <none>
I20260812 00:15:44.277806 32479 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:44.277879 32479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:44.278045 32479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:44.279156 32479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "eb9d8613d9b04fcca1189a4c2bdc504a"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "7e74d95f2cbc478932d133dbc5593aa7"
server_key_iv: "e0f7ae13772a8808a918558385128ebd"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.279534 32479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "eb9d8613d9b04fcca1189a4c2bdc504a"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "7e74d95f2cbc478932d133dbc5593aa7"
server_key_iv: "e0f7ae13772a8808a918558385128ebd"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.280844 32479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:44.281601 32504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.281778 32479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:15:44.281886 32479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "eb9d8613d9b04fcca1189a4c2bdc504a"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "7e74d95f2cbc478932d133dbc5593aa7"
server_key_iv: "e0f7ae13772a8808a918558385128ebd"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.281970 32479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:44.297926 32479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:44.298214 32479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:44.298370 32479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:44.298594 32479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:44.298923 32479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:44.298978 32479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.299019 32479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:44.299047 32479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.304981 32479 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:43889
I20260812 00:15:44.305052 32714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:43889 every 8 connection(s)
I20260812 00:15:44.305411 32479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:15:44.307008 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 32479
I20260812 00:15:44.307121 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:15:44.307322 32652 external_mini_cluster.cc:1428] Setting key 545ef37506966da318fb19f1ef73108d
I20260812 00:15:44.308444 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45007
--builtin_ntp_servers=127.31.227.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 00:15:44.311422 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.306147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:53547 (local address 127.31.227.62:45007)
0812 00:15:44.306200 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.306203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.306474 (+ 271us) server_negotiation.cc:407] Connection header received
0812 00:15:44.306986 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.306987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.307001 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.307067 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:44.307997 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.308764 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.309819 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.310144 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.310238 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.311177 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:44.311179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:44.311179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:44.311183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:44.311196 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:44.311200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:44.311243 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.311344 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.311366 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:15:44.311672 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.306140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45007 (local address 127.31.227.2:53547)
0812 00:15:44.306345 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.306379 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.306859 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.307139 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.307143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.307151 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:44.307874 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.307881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.308868 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.308869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.309743 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.309746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.310230 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.310231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.310428 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.311018 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:44.311019 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:44.311058 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:44.311088 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:44.311257 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:44.311262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:44.311265 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.311341 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.311459 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260812 00:15:44.312201 32717 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45007
I20260812 00:15:44.312278 32717 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:44.312485 32717 heartbeater.cc:511] Master 127.31.227.62:45007 requested a full tablet report, sending...
I20260812 00:15:44.312870 32149 ts_manager.cc:194] Registered new tserver with Master: eb9d8613d9b04fcca1189a4c2bdc504a (127.31.227.2:43889)
I20260812 00:15:44.313418 32149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.2:53547
W20260812 00:15:44.393455 32721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:44.393673 32721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:44.393695 32721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:44.394763 32721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:44.394846 32721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:44.394892 32721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:44.396236 32721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:44.396433 32721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:44.396682 32721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:44.401731 32732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:44.401731 32721 server_base.cc:1034] running on GCE node
W20260812 00:15:44.401783 32730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:44.401787 32734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:44.402128 32721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:44.402391 32721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:44.403553 32721 hybrid_clock.cc:648] HybridClock initialized: now 1786493744403550 us; error 35 us; skew 500 ppm
I20260812 00:15:44.404835 32721 webserver.cc:460] Webserver started at http://127.31.227.3:35181/ using document root <none> and password file <none>
I20260812 00:15:44.405123 32721 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:44.405196 32721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:44.405366 32721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:44.406505 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "a351ffed13f6411fb6bb1d6e4c663698"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "8153dfee1e04ee0d646cff18e76f6a7f"
server_key_iv: "1b480f8c3d53065fbcd4c2984d3a91ed"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.406872 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "a351ffed13f6411fb6bb1d6e4c663698"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "8153dfee1e04ee0d646cff18e76f6a7f"
server_key_iv: "1b480f8c3d53065fbcd4c2984d3a91ed"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.408172 32721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:44.408895 32742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.409075 32721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:44.409175 32721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "a351ffed13f6411fb6bb1d6e4c663698"
format_stamp: "Formatted at 2026-08-12 00:15:44 on dist-test-slave-zr1t"
server_key: "8153dfee1e04ee0d646cff18e76f6a7f"
server_key_iv: "1b480f8c3d53065fbcd4c2984d3a91ed"
server_key_version: "encryptionkey@0"
I20260812 00:15:44.409264 32721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:44.419638 32721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:44.419957 32721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:44.420109 32721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:44.420329 32721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:44.420665 32721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:44.420727 32721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.420769 32721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:44.420797 32721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:44.426971 32721 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41013
I20260812 00:15:44.427018 459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41013 every 8 connection(s)
I20260812 00:15:44.427366 32721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:15:44.432447 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.428139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:47931 (local address 127.31.227.62:45007)
0812 00:15:44.428204 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.428206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.428451 (+ 245us) server_negotiation.cc:407] Connection header received
0812 00:15:44.428948 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.428949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.428960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.429014 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:44.429750 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.430245 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.431206 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.431384 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.431433 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.432211 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:44.432212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:44.432212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:44.432217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:44.432231 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:44.432235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:44.432278 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.432360 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.432386 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:15:44.432673 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.428129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45007 (local address 127.31.227.3:47931)
0812 00:15:44.428329 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.428364 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.428860 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.429066 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.429070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.429078 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:44.429674 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.429679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.430328 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.430328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.431132 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.431134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.431460 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.431461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.431569 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.432082 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:44.432082 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:44.432109 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:44.432132 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:44.432289 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:44.432293 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:44.432296 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.432348 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.432429 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260812 00:15:44.433303 463 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45007
I20260812 00:15:44.433431 463 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:44.433672 463 heartbeater.cc:511] Master 127.31.227.62:45007 requested a full tablet report, sending...
I20260812 00:15:44.434026 32149 ts_manager.cc:194] Registered new tserver with Master: a351ffed13f6411fb6bb1d6e4c663698 (127.31.227.3:41013)
I20260812 00:15:44.434397 32149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.3:47931
I20260812 00:15:44.435544 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 32721
I20260812 00:15:44.435662 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:15:44.435887 32652 external_mini_cluster.cc:1428] Setting key ab79f5c4342ec4274e46d532cd454055
I20260812 00:15:44.436858 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:44.441023 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.438304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.31.227.62:45007)
0812 00:15:44.438371 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.438372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.438424 (+ 52us) server_negotiation.cc:407] Connection header received
0812 00:15:44.438483 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.438484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.438494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.438543 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:44.438696 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.439153 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.439836 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.440012 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.440061 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.440752 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:44.440753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:44.440753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:44.440757 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:44.440768 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:44.440772 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:44.440839 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.440904 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.440914 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:15:44.442668 32149 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:44.444218 32149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:44.450712 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.446400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41013 (local address 127.0.0.1:36796)
0812 00:15:44.446559 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.446599 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.446671 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.447175 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.447179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.447181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:44.447253 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.447255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.448606 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.448607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.449681 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.449682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.450398 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.450400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.450488 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.450501 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.450576 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.450633 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":4,"thread_start_us":92,"threads_started":1}
I20260812 00:15:44.450727 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.446795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.31.227.3:41013)
0812 00:15:44.446979 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.446983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.446992 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:44.447033 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.447035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.447052 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.447336 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:44.447409 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.448520 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.449770 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.450308 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.450374 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.450479 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.450610 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.450649 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260812 00:15:44.450745 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.446531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.31.227.2:43889)
0812 00:15:44.446818 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.446828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.447108 (+ 280us) server_negotiation.cc:407] Connection header received
0812 00:15:44.447242 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.447245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.447261 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.447353 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:44.447595 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.448651 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.449853 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.450331 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.450429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.450519 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.450614 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.450662 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260812 00:15:44.450984 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.446407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:43889 (local address 127.0.0.1:59296)
0812 00:15:44.447033 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.447045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.447140 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.447464 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.447465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.447467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:44.447529 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.447531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.448750 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.448752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.449710 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.449713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.450448 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.450449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.450516 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.450520 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.450871 (+ 351us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.450915 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":94,"threads_started":1}
I20260812 00:15:44.451793 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.446648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:38741 (local address 127.0.0.1:41352)
0812 00:15:44.447083 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.447094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.447144 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.447411 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.447412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.447413 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:44.447449 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.447451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.448346 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.448348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.449679 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.449682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.451545 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.451546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.451633 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.451637 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.451689 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.451727 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":72,"threads_started":1}
I20260812 00:15:44.451818 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.446771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.31.227.1:38741)
0812 00:15:44.446975 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.446979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.447201 (+ 222us) server_negotiation.cc:407] Connection header received
0812 00:15:44.447252 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.447254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.447270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.447341 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:44.447535 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.448254 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.450519 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.451283 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.451417 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.451511 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.451705 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.451742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260812 00:15:44.451751 32584 tablet_service.cc:1467] Processing CreateTablet for tablet 35031a217f394d21bb980426f572c205 (DEFAULT_TABLE table=test-table [id=1bd087c4a39641f9b2b522dc34435f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:44.451761 354 tablet_service.cc:1467] Processing CreateTablet for tablet 35031a217f394d21bb980426f572c205 (DEFAULT_TABLE table=test-table [id=1bd087c4a39641f9b2b522dc34435f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:44.452142 32584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35031a217f394d21bb980426f572c205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:44.452142 354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35031a217f394d21bb980426f572c205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:44.455757 489 tablet_bootstrap.cc:492] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Bootstrap starting.
I20260812 00:15:44.456243 32356 tablet_service.cc:1467] Processing CreateTablet for tablet 35031a217f394d21bb980426f572c205 (DEFAULT_TABLE table=test-table [id=1bd087c4a39641f9b2b522dc34435f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:44.456475 489 tablet_bootstrap.cc:654] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:44.456622 32356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35031a217f394d21bb980426f572c205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:44.456866 489 log.cc:824] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:44.457181 490 tablet_bootstrap.cc:492] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Bootstrap starting.
I20260812 00:15:44.457613 489 tablet_bootstrap.cc:492] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: No bootstrap required, opened a new log
I20260812 00:15:44.457683 489 ts_tablet_manager.cc:1397] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:44.457970 490 tablet_bootstrap.cc:654] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:44.458261 490 log.cc:824] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:44.459049 490 tablet_bootstrap.cc:492] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: No bootstrap required, opened a new log
I20260812 00:15:44.459129 490 ts_tablet_manager.cc:1397] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:44.460242 496 tablet_bootstrap.cc:492] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b: Bootstrap starting.
I20260812 00:15:44.460502 489 raft_consensus.cc:348] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.460635 489 raft_consensus.cc:374] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:44.460691 489 raft_consensus.cc:729] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9d8613d9b04fcca1189a4c2bdc504a, State: Initialized, Role: FOLLOWER
I20260812 00:15:44.460812 489 consensus_queue.cc:260] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.461090 489 ts_tablet_manager.cc:1428] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:44.461117 32717 heartbeater.cc:503] Master 127.31.227.62:45007 was elected leader, sending a full tablet report...
I20260812 00:15:44.461184 496 tablet_bootstrap.cc:654] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:44.461591 496 log.cc:824] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:44.462275 490 raft_consensus.cc:348] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.462477 496 tablet_bootstrap.cc:492] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b: No bootstrap required, opened a new log
I20260812 00:15:44.462483 490 raft_consensus.cc:374] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:44.462529 490 raft_consensus.cc:729] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a351ffed13f6411fb6bb1d6e4c663698, State: Initialized, Role: FOLLOWER
I20260812 00:15:44.462553 496 ts_tablet_manager.cc:1397] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 00:15:44.462646 490 consensus_queue.cc:260] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.462908 490 ts_tablet_manager.cc:1428] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 00:15:44.463066 463 heartbeater.cc:503] Master 127.31.227.62:45007 was elected leader, sending a full tablet report...
I20260812 00:15:44.464624 496 raft_consensus.cc:348] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.464771 496 raft_consensus.cc:374] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:44.464818 496 raft_consensus.cc:729] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144ec070138848e18e6924863f1fe56b, State: Initialized, Role: FOLLOWER
I20260812 00:15:44.464929 496 consensus_queue.cc:260] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.465179 496 ts_tablet_manager.cc:1428] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:15:44.465276 32474 heartbeater.cc:503] Master 127.31.227.62:45007 was elected leader, sending a full tablet report...
I20260812 00:15:44.476281 504 raft_consensus.cc:482] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:44.476397 504 raft_consensus.cc:504] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.476817 504 leader_election.cc:288] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a351ffed13f6411fb6bb1d6e4c663698 (127.31.227.3:41013), eb9d8613d9b04fcca1189a4c2bdc504a (127.31.227.2:43889)
I20260812 00:15:44.479743 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.477064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:34587 (local address 127.31.227.3:41013)
0812 00:15:44.477125 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.477127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.477138 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:15:44.477169 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.477170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.477179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.477265 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:44.477429 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.478033 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.478990 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.479370 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.479471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.479479 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.479633 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.479672 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:15:44.479753 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.476958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41013 (local address 127.31.227.1:34587)
0812 00:15:44.477023 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.477038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.477087 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.477271 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.477273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.477275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:44.477343 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.477345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.478125 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.478126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.478881 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.478883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.479473 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.479474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.479557 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.479560 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.479626 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.479672 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":10}
I20260812 00:15:44.479748 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.477247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:54751 (local address 127.31.227.2:43889)
0812 00:15:44.477297 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.477300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.477309 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:44.477364 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.477365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.477373 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.477446 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:44.477604 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.478066 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.479007 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.479384 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.479467 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.479476 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.479637 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.479674 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:15:44.479770 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.477054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:43889 (local address 127.31.227.1:54751)
0812 00:15:44.477243 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.477252 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:44.477297 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:44.477449 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:44.477450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:44.477452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:44.477548 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.477550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.478150 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.478152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.478911 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:44.478914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.479479 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.479481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:44.479564 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.479567 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:44.479632 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:15:44.479674 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260812 00:15:44.480116 382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35031a217f394d21bb980426f572c205" candidate_uuid: "144ec070138848e18e6924863f1fe56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a351ffed13f6411fb6bb1d6e4c663698" is_pre_election: true
I20260812 00:15:44.480285 382 raft_consensus.cc:2454] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 144ec070138848e18e6924863f1fe56b in term 0.
I20260812 00:15:44.480356 32620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35031a217f394d21bb980426f572c205" candidate_uuid: "144ec070138848e18e6924863f1fe56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" is_pre_election: true
I20260812 00:15:44.480522 32620 raft_consensus.cc:2454] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 144ec070138848e18e6924863f1fe56b in term 0.
I20260812 00:15:44.480528 32283 leader_election.cc:302] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144ec070138848e18e6924863f1fe56b, a351ffed13f6411fb6bb1d6e4c663698; no voters:
I20260812 00:15:44.480654 504 raft_consensus.cc:2781] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:44.480702 504 raft_consensus.cc:482] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:44.480731 504 raft_consensus.cc:3037] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:44.481465 504 raft_consensus.cc:504] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.481647 504 leader_election.cc:288] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [CANDIDATE]: Term 1 election: Requested vote from peers a351ffed13f6411fb6bb1d6e4c663698 (127.31.227.3:41013), eb9d8613d9b04fcca1189a4c2bdc504a (127.31.227.2:43889)
I20260812 00:15:44.481773 382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35031a217f394d21bb980426f572c205" candidate_uuid: "144ec070138848e18e6924863f1fe56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a351ffed13f6411fb6bb1d6e4c663698"
I20260812 00:15:44.481853 382 raft_consensus.cc:3037] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:44.482592 382 raft_consensus.cc:2454] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 144ec070138848e18e6924863f1fe56b in term 1.
I20260812 00:15:44.482758 32283 leader_election.cc:302] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144ec070138848e18e6924863f1fe56b, a351ffed13f6411fb6bb1d6e4c663698; no voters:
I20260812 00:15:44.482772 32620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35031a217f394d21bb980426f572c205" candidate_uuid: "144ec070138848e18e6924863f1fe56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a"
I20260812 00:15:44.482863 504 raft_consensus.cc:2781] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:44.482843 32620 raft_consensus.cc:3037] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:44.483018 504 raft_consensus.cc:686] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [term 1 LEADER]: Becoming Leader. State: Replica: 144ec070138848e18e6924863f1fe56b, State: Running, Role: LEADER
I20260812 00:15:44.483122 504 consensus_queue.cc:237] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } }
I20260812 00:15:44.483875 32620 raft_consensus.cc:2454] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 144ec070138848e18e6924863f1fe56b in term 1.
I20260812 00:15:44.484053 32149 catalog_manager.cc:5495] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b reported cstate change: term changed from 0 to 1, leader changed from <none> to 144ec070138848e18e6924863f1fe56b (127.31.227.1). New cstate: current_term: 1 leader_uuid: "144ec070138848e18e6924863f1fe56b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "144ec070138848e18e6924863f1fe56b" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38741 } health_report { overall_health: HEALTHY } } }
I20260812 00:15:44.495784 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.493518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.31.227.1:38741)
0812 00:15:44.493603 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:44.493604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:44.493613 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:44.493643 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:44.493644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:44.493661 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:44.493720 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:44.493835 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.494280 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.495045 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:44.495224 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:44.495277 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:44.495437 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:44.495525 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:44.495593 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:44.495645 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 00:15:44.495655 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 00:15:44.497670 382 raft_consensus.cc:1261] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 1 FOLLOWER]: Refusing update from remote peer 144ec070138848e18e6924863f1fe56b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:44.497670 32620 raft_consensus.cc:1261] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 1 FOLLOWER]: Refusing update from remote peer 144ec070138848e18e6924863f1fe56b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:44.498028 510 consensus_queue.cc:1035] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a351ffed13f6411fb6bb1d6e4c663698" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:44.498162 504 consensus_queue.cc:1035] T 35031a217f394d21bb980426f572c205 P 144ec070138848e18e6924863f1fe56b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9d8613d9b04fcca1189a4c2bdc504a" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:44.500608 518 mvcc.cc:205] Tried to move back new op lower bound from 7317478377459699712 to 7317478377403559936. Current Snapshot: MvccSnapshot[applied={T|T < 7317478377459699712}]
I20260812 00:15:44.501336 519 mvcc.cc:205] Tried to move back new op lower bound from 7317478377459699712 to 7317478377403559936. Current Snapshot: MvccSnapshot[applied={T|T < 7317478377459699712}]
I20260812 00:15:44.503708 32147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:44.503793 32147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:44.504602 32147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3a66105f09545e38a58740ec4db67b1: Sending DeleteTablet for 3 replicas of tablet 35031a217f394d21bb980426f572c205
I20260812 00:15:44.504972 354 tablet_service.cc:1514] Processing DeleteTablet for tablet 35031a217f394d21bb980426f572c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:44 UTC) from {username='slave'} at 127.0.0.1:36796
I20260812 00:15:44.505211 532 tablet_replica.cc:330] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: stopping tablet replica
I20260812 00:15:44.505232 32584 tablet_service.cc:1514] Processing DeleteTablet for tablet 35031a217f394d21bb980426f572c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:44 UTC) from {username='slave'} at 127.0.0.1:59296
I20260812 00:15:44.505326 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 32255
I20260812 00:15:44.505321 532 raft_consensus.cc:2229] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:44.505458 532 raft_consensus.cc:2258] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:44.505436 32356 tablet_service.cc:1514] Processing DeleteTablet for tablet 35031a217f394d21bb980426f572c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:44 UTC) from {username='slave'} at 127.0.0.1:41352
I20260812 00:15:44.509307 532 ts_tablet_manager.cc:1905] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:44.511286 533 tablet_replica.cc:330] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: stopping tablet replica
I20260812 00:15:44.511408 533 raft_consensus.cc:2229] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:44.511523 533 raft_consensus.cc:2258] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:44.513271 532 ts_tablet_manager.cc:1918] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:44.513360 532 log.cc:1196] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/35031a217f394d21bb980426f572c205
I20260812 00:15:44.513433 533 ts_tablet_manager.cc:1905] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:44.513664 532 ts_tablet_manager.cc:1939] T 35031a217f394d21bb980426f572c205 P a351ffed13f6411fb6bb1d6e4c663698: Deleting consensus metadata
I20260812 00:15:44.514137 32127 catalog_manager.cc:4841] TS a351ffed13f6411fb6bb1d6e4c663698 (127.31.227.3:41013): tablet 35031a217f394d21bb980426f572c205 (table test-table [id=1bd087c4a39641f9b2b522dc34435f44]) successfully deleted
I20260812 00:15:44.514927 533 ts_tablet_manager.cc:1918] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:44.514988 533 log.cc:1196] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/35031a217f394d21bb980426f572c205
I20260812 00:15:44.515277 533 ts_tablet_manager.cc:1939] T 35031a217f394d21bb980426f572c205 P eb9d8613d9b04fcca1189a4c2bdc504a: Deleting consensus metadata
I20260812 00:15:44.515707 32126 catalog_manager.cc:4841] TS eb9d8613d9b04fcca1189a4c2bdc504a (127.31.227.2:43889): tablet 35031a217f394d21bb980426f572c205 (table test-table [id=1bd087c4a39641f9b2b522dc34435f44]) successfully deleted
W20260812 00:15:44.520275 32127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:38741 (error 108)
I20260812 00:15:44.520546 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 32479
I20260812 00:15:44.520815 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:44.520686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:38741 (local address 127.0.0.1:41364)
0812 00:15:44.520739 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:44.520776 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:38741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:15:44.520941 32127 catalog_manager.cc:4584] TS 144ec070138848e18e6924863f1fe56b (127.31.227.1:38741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35031a217f394d21bb980426f572c205: Network error: Client connection negotiation failed: client connection to 127.31.227.1:38741: connect: Connection refused (error 111)
I20260812 00:15:44.532796 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 32721
I20260812 00:15:44.546494 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 32106
2026-08-12T00:15:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (724 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:44 dist-test-slave-zr1t krb5kdc[540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:44 dist-test-slave-zr1t krb5kdc[540](info): set up 2 sockets
Aug 12 00:15:44 dist-test-slave-zr1t krb5kdc[540](info): commencing operation
krb5kdc: starting...
W20260812 00:15:45.806492 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:45 dist-test-slave-zr1t krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:15:45.924130 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45865
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:45865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:46.007468 560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:46.007671 560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:46.007694 560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:46.008761 560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:46.008798 560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:46.008812 560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:46.008826 560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:46.008837 560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:46.010344 560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:45865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:45865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:46.010548 560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:46.010730 560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:46.015620 567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:46.015607 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:46.015743 560 server_base.cc:1034] running on GCE node
W20260812 00:15:46.015586 569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:46.016033 560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:46.016378 560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:46.017596 560 hybrid_clock.cc:648] HybridClock initialized: now 1786493746017567 us; error 42 us; skew 500 ppm
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:46.020222 560 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:15:46.020674 560 webserver.cc:460] Webserver started at http://127.31.227.62:34547/ using document root <none> and password file <none>
I20260812 00:15:46.020915 560 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:46.020992 560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:46.021157 560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:46.022311 560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "05c5d9f3d2e6495dabdeb38c50e701f0"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "337b5c7af03db7cdfe1c1e2fc1add908"
server_key_iv: "b17fb11fd34b055dfa1ccfc4cfab4f7f"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.022681 560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "05c5d9f3d2e6495dabdeb38c50e701f0"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "337b5c7af03db7cdfe1c1e2fc1add908"
server_key_iv: "b17fb11fd34b055dfa1ccfc4cfab4f7f"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.024063 560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:15:46.024806 593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.024995 560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:15:46.025104 560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "05c5d9f3d2e6495dabdeb38c50e701f0"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "337b5c7af03db7cdfe1c1e2fc1add908"
server_key_iv: "b17fb11fd34b055dfa1ccfc4cfab4f7f"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.025179 560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:46.034017 560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:46.035985 560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:46.036154 560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:46.040539 560 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:45865
I20260812 00:15:46.040575 692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:45865 every 8 connection(s)
I20260812 00:15:46.040951 560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:46.041971 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:46.044373 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: Bootstrap starting.
I20260812 00:15:46.044924 693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:46.045163 693 log.cc:824] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:46.045910 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: No bootstrap required, opened a new log
I20260812 00:15:46.047499 693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45865 } }
I20260812 00:15:46.047642 693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:46.047665 693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c5d9f3d2e6495dabdeb38c50e701f0, State: Initialized, Role: FOLLOWER
I20260812 00:15:46.047811 693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45865 } }
I20260812 00:15:46.047899 693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:46.047922 693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:46.047951 693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:46.048480 693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45865 } }
I20260812 00:15:46.048570 693 leader_election.cc:302] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05c5d9f3d2e6495dabdeb38c50e701f0; no voters:
I20260812 00:15:46.048712 693 leader_election.cc:288] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:46.048779 696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:46.048951 696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [term 1 LEADER]: Becoming Leader. State: Replica: 05c5d9f3d2e6495dabdeb38c50e701f0, State: Running, Role: LEADER
I20260812 00:15:46.048995 693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:46.049043 696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45865 } }
I20260812 00:15:46.049376 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 560
I20260812 00:15:46.049500 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:46.049795 32652 external_mini_cluster.cc:1428] Setting key 19517650da179de7d4363405eb87f322
I20260812 00:15:46.050181 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45865 } } }
I20260812 00:15:46.050314 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [sys.catalog]: This master's current role is: LEADER
W20260812 00:15:46.051143 720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:46.051198 720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:46.051167 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05c5d9f3d2e6495dabdeb38c50e701f0. Latest consensus state: current_term: 1 leader_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c5d9f3d2e6495dabdeb38c50e701f0" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 45865 } } }
I20260812 00:15:46.051235 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:46.051389 717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:46.051610 717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:46.053061 717 catalog_manager.cc:1347] Generated new cluster ID: 07f586e1fd5c48f0b4e3b980036c95f7
I20260812 00:15:46.053135 717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:46.059680 717 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:46.060410 717 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:46.062511 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.050515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.31.227.62:45865)
0812 00:15:46.050748 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.050758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.051103 (+ 345us) server_negotiation.cc:407] Connection header received
0812 00:15:46.051230 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.051238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.051301 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.051452 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:46.052244 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.053684 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.054763 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.055395 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.055527 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.058548 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:46.058552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:46.058553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:46.058566 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:46.060733 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.061150 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.061153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.061153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.061263 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.061536 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.061537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.061537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.061837 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:46.062005 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.062115 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.062265 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260812 00:15:46.064769 717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: Generated new TSK 0
I20260812 00:15:46.064950 717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.227.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
Entry for principal kudu/127.31.227.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab.
I20260812 00:15:46.093706 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45865
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:46.176422 731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:46.176635 731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:46.176659 731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:46.177886 731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:46.177925 731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:46.177986 731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:46.179440 731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.1.keytab
--principal=kudu/127.31.227.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:46.179682 731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:46.179905 731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:46.185037 744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:46.185065 745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:46.185132 731 server_base.cc:1034] running on GCE node
W20260812 00:15:46.185037 747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:46.185420 731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:46.185724 731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:46.186879 731 hybrid_clock.cc:648] HybridClock initialized: now 1786493746186868 us; error 30 us; skew 500 ppm
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:46.189399 731 init.cc:377] Logged in from keytab as kudu/127.31.227.1@KRBTEST.COM (short username kudu)
I20260812 00:15:46.189850 731 webserver.cc:460] Webserver started at http://127.31.227.1:46335/ using document root <none> and password file <none>
I20260812 00:15:46.190042 731 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:46.190084 731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:46.190176 731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:46.191165 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "ecc74799e40a4f75a6edb9538652f5b3"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "99fbb8ff6dbff470571421144a669304"
server_key_iv: "05f829859d9ccab6394b23763c4c4faa"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.191468 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "ecc74799e40a4f75a6edb9538652f5b3"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "99fbb8ff6dbff470571421144a669304"
server_key_iv: "05f829859d9ccab6394b23763c4c4faa"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.192705 731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:46.193380 755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.193579 731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:15:46.193666 731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "ecc74799e40a4f75a6edb9538652f5b3"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "99fbb8ff6dbff470571421144a669304"
server_key_iv: "05f829859d9ccab6394b23763c4c4faa"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.193722 731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:46.211505 731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:46.213340 731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:46.213459 731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:46.213752 731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:46.214075 731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:46.214109 731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.214131 731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:46.214181 731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.220096 731 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:44519
I20260812 00:15:46.220194 967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:44519 every 8 connection(s)
I20260812 00:15:46.220504 731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.1@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:46.228559 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 731
I20260812 00:15:46.228704 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:15:46.228984 32652 external_mini_cluster.cc:1428] Setting key b3d192d54795de5a7d3e0b3e604cb92e
I20260812 00:15:46.229934 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.221283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:49637 (local address 127.31.227.62:45865)
0812 00:15:46.221350 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.221352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.221588 (+ 236us) server_negotiation.cc:407] Connection header received
0812 00:15:46.222085 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.222086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.222096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.222153 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:46.223063 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.223564 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.224506 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.224697 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.224748 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.226955 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:46.226956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:46.226956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:46.226962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:46.228268 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.228704 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.228705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.228706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.228789 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.229115 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.229116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.229116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.229295 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:46.229365 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.229759 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.229864 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:15:46.230188 973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.221265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45865 (local address 127.31.227.1:49637)
0812 00:15:46.221433 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.221477 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.221997 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.222205 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.222208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.222783 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:46.222983 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.222988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.223645 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.223645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.224426 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.224428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.224770 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.224771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.224869 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.225404 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:46.225413 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:46.226867 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:46.228339 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:46.228345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:46.228347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:46.228620 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:46.228855 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:46.228857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:46.228857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:46.229012 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:46.229368 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:46.229376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:46.229578 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.229719 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.229916 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260812 00:15:46.230685 969 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45865
I20260812 00:15:46.230793 969 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:46.230984 969 heartbeater.cc:511] Master 127.31.227.62:45865 requested a full tablet report, sending...
I20260812 00:15:46.231510 628 ts_manager.cc:194] Registered new tserver with Master: ecc74799e40a4f75a6edb9538652f5b3 (127.31.227.1:44519)
I20260812 00:15:46.232439 628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.1@KRBTEST.COM'} at 127.31.227.1:49637
WARNING: no policy specified for kudu/127.31.227.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
Entry for principal kudu/127.31.227.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab.
I20260812 00:15:46.254128 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45865
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:46.336854 981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:46.337046 981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:46.337069 981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:46.338114 981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:46.338150 981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:46.338174 981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:46.339470 981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.2.keytab
--principal=kudu/127.31.227.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:46.339733 981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:46.339952 981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:46.344921 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:46.344933 988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:46.344938 989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:46.344964 981 server_base.cc:1034] running on GCE node
I20260812 00:15:46.345350 981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:46.345629 981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:46.346771 981 hybrid_clock.cc:648] HybridClock initialized: now 1786493746346761 us; error 27 us; skew 500 ppm
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:46.349251 981 init.cc:377] Logged in from keytab as kudu/127.31.227.2@KRBTEST.COM (short username kudu)
I20260812 00:15:46.349767 981 webserver.cc:460] Webserver started at http://127.31.227.2:45239/ using document root <none> and password file <none>
I20260812 00:15:46.350013 981 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:46.350059 981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:46.350229 981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:46.351294 981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "84c1ad24526143c3a2aee26fa4fb24a4"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "6a86520b3dbf8a8390e1aadfec1b69d2"
server_key_iv: "ac60e82e3bfe7d16f663d495a40da800"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.351662 981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "84c1ad24526143c3a2aee26fa4fb24a4"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "6a86520b3dbf8a8390e1aadfec1b69d2"
server_key_iv: "ac60e82e3bfe7d16f663d495a40da800"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.352974 981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:15:46.353741 1001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.353930 981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:46.354023 981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "84c1ad24526143c3a2aee26fa4fb24a4"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "6a86520b3dbf8a8390e1aadfec1b69d2"
server_key_iv: "ac60e82e3bfe7d16f663d495a40da800"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.354077 981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:46.365689 981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:46.367427 981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:46.367547 981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:46.367719 981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:46.367964 981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:46.367992 981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.368012 981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:46.368024 981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.374039 981 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:40559
I20260812 00:15:46.374117 1208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:40559 every 8 connection(s)
I20260812 00:15:46.374439 981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:15:46.379355 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 981
I20260812 00:15:46.379451 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:15:46.379657 32652 external_mini_cluster.cc:1428] Setting key 40ac78211795a0a9bacb80f5c63143f8
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.2@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:46.383899 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.375200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:43597 (local address 127.31.227.62:45865)
0812 00:15:46.375271 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.375273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.375487 (+ 214us) server_negotiation.cc:407] Connection header received
0812 00:15:46.375958 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.375959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.375969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.376026 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:46.376919 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.377413 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.378394 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.378578 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.378631 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.380967 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:46.380969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:46.380969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:46.380978 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:46.382479 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.382911 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.382913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.382914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.382979 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.383233 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.383235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.383235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.383362 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:46.383441 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.383724 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.383808 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:15:46.384104 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.375159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45865 (local address 127.31.227.2:43597)
0812 00:15:46.375353 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.375393 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.375869 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.376077 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.376080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.376641 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:46.376827 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.376832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.377496 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.377497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.378320 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.378322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.378653 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.378654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.378753 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.379277 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:46.379285 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:46.380858 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:46.382585 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:46.382593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:46.382595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:46.382819 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:46.383055 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:46.383055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:46.383056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:46.383163 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:46.383451 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:46.383458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:46.383593 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.383702 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.383863 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260812 00:15:46.384711 1209 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45865
I20260812 00:15:46.384819 1209 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:46.385012 1209 heartbeater.cc:511] Master 127.31.227.62:45865 requested a full tablet report, sending...
I20260812 00:15:46.385499 628 ts_manager.cc:194] Registered new tserver with Master: 84c1ad24526143c3a2aee26fa4fb24a4 (127.31.227.2:40559)
I20260812 00:15:46.386075 628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.2@KRBTEST.COM'} at 127.31.227.2:43597
WARNING: no policy specified for kudu/127.31.227.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
Entry for principal kudu/127.31.227.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab.
I20260812 00:15:46.405448 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45865
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:46.489280 1216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:46.489494 1216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:46.489578 1216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:46.490736 1216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:46.490779 1216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:46.490805 1216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:46.492329 1216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.3.keytab
--principal=kudu/127.31.227.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:46.492592 1216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:46.492841 1216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:15:46.498108 1216 server_base.cc:1034] running on GCE node
W20260812 00:15:46.498070 1226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:46.498095 1230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:46.498071 1227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:46.498473 1216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:46.498747 1216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:46.499918 1216 hybrid_clock.cc:648] HybridClock initialized: now 1786493746499911 us; error 34 us; skew 500 ppm
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:46.502638 1216 init.cc:377] Logged in from keytab as kudu/127.31.227.3@KRBTEST.COM (short username kudu)
I20260812 00:15:46.503087 1216 webserver.cc:460] Webserver started at http://127.31.227.3:39365/ using document root <none> and password file <none>
I20260812 00:15:46.503325 1216 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:46.503396 1216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:46.503557 1216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:46.504604 1216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "9e3e5271bbad45d7aab2a9f4c14aa604"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "7c20905e054221ad8c2404d88432b29f"
server_key_iv: "e5d5ee6bfef8251bcb341bd4857c53a4"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.504964 1216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "9e3e5271bbad45d7aab2a9f4c14aa604"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "7c20905e054221ad8c2404d88432b29f"
server_key_iv: "e5d5ee6bfef8251bcb341bd4857c53a4"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.506379 1216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:15:46.507133 1248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.507306 1216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:46.507416 1216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "9e3e5271bbad45d7aab2a9f4c14aa604"
format_stamp: "Formatted at 2026-08-12 00:15:46 on dist-test-slave-zr1t"
server_key: "7c20905e054221ad8c2404d88432b29f"
server_key_iv: "e5d5ee6bfef8251bcb341bd4857c53a4"
server_key_version: "encryptionkey@0"
I20260812 00:15:46.507491 1216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:46.537186 1216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:46.539758 1216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:46.539959 1216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:46.540268 1216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:46.540613 1216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:46.540673 1216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.540714 1216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:46.540743 1216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:46.547432 1216 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:41835
I20260812 00:15:46.547503 1437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:41835 every 8 connection(s)
I20260812 00:15:46.547906 1216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:15:46.551055 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1216
I20260812 00:15:46.551177 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:15:46.551366 32652 external_mini_cluster.cc:1428] Setting key 560aba742f680b87a60e2ef2ae1898b5
Aug 12 00:15:46 dist-test-slave-zr1t krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.3@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:15:46.557673 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.548725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:43873 (local address 127.31.227.62:45865)
0812 00:15:46.548794 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.548797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.549017 (+ 220us) server_negotiation.cc:407] Connection header received
0812 00:15:46.549592 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.549593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.549607 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.549693 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:46.550762 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.551548 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.552720 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.552933 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.553028 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.555206 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:46.555207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:46.555208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:46.555214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:46.556375 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.556826 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.556827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.556827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.556878 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.557107 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.557108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.557108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.557213 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:46.557264 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.557494 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.557607 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:15:46.557869 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.548687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:45865 (local address 127.31.227.3:43873)
0812 00:15:46.548863 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.548909 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.549443 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.549775 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.549780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.550437 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:46.550645 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.550651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.551672 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.551673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.552629 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.552633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.553041 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.553042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.553178 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.553706 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:15:46.553714 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:46.555118 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:46.556446 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:46.556453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:46.556455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:46.556763 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:46.556938 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:15:46.556939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:15:46.556939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:15:46.557049 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:15:46.557267 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:46.557272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:46.557386 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.557467 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.557651 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260812 00:15:46.558344 1438 heartbeater.cc:347] Connected to a master server at 127.31.227.62:45865
I20260812 00:15:46.558459 1438 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:46.558645 1438 heartbeater.cc:511] Master 127.31.227.62:45865 requested a full tablet report, sending...
I20260812 00:15:46.559013 628 ts_manager.cc:194] Registered new tserver with Master: 9e3e5271bbad45d7aab2a9f4c14aa604 (127.31.227.3:41835)
I20260812 00:15:46.559381 628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.227.3@KRBTEST.COM'} at 127.31.227.3:43873
I20260812 00:15:46.562984 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:46.570367 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.564299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.31.227.62:45865)
0812 00:15:46.564354 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.564356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.564473 (+ 117us) server_negotiation.cc:407] Connection header received
0812 00:15:46.564511 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.564512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.564520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.564568 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:46.565267 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.565809 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.566534 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.566717 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.566800 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.568034 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:46.568036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:46.568036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:46.568043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:46.569202 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.569553 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.569554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.569554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.569626 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:46.569837 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:46.569838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:46.569838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:46.569984 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:46.570090 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.570193 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.570270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:15:46.571825 628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:46.573374 628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:46.579888 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.575644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41835 (local address 127.0.0.1:57366)
0812 00:15:46.575757 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.575784 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.575857 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.576181 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.576186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.576193 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:46.576273 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.576275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.577502 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.577503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.578823 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.578826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.579543 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.579638 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.579651 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.579767 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.579820 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":76,"threads_started":1}
I20260812 00:15:46.579897 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.575753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.31.227.3:41835)
0812 00:15:46.575945 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.575949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.575958 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:46.575997 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.575998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.576009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.576098 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:46.576364 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.577368 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.578925 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.579454 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579541 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.579600 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.579785 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.579827 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260812 00:15:46.580225 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.576035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44519 (local address 127.0.0.1:56740)
0812 00:15:46.576264 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.576375 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.576420 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.576939 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.576940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.576942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:46.576983 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.576984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.578051 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.578052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.578960 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.578963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.579944 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.580051 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.580056 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.580109 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.580149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260812 00:15:46.580274 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.576182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.31.227.1:44519)
0812 00:15:46.576782 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.576786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.576795 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:46.576835 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.576837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.576850 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.576926 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:46.577044 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.577942 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579078 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.579770 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.579892 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.580140 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.580190 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":74,"threads_started":1}
I20260812 00:15:46.580430 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.575858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:40559 (local address 127.0.0.1:48154)
0812 00:15:46.575966 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.575976 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.576021 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.576647 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.576647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.576649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:46.576690 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.576691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.577801 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.577802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.578948 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.578950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.580071 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.580073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.580130 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.580133 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.580179 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.580211 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260812 00:15:46.580616 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.576284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.31.227.2:40559)
0812 00:15:46.576489 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.576494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.576503 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:46.576544 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.576545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.576560 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.576642 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:46.576766 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.577687 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579031 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.579770 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.579840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.579892 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.580269 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.580560 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260812 00:15:46.580770 1322 tablet_service.cc:1467] Processing CreateTablet for tablet bd815e00bb504409989fcf136f34f4a9 (DEFAULT_TABLE table=test-table [id=6f249e3f4e174a2b9c292e06be49d969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:46.581110 1322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd815e00bb504409989fcf136f34f4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:46.581248 1093 tablet_service.cc:1467] Processing CreateTablet for tablet bd815e00bb504409989fcf136f34f4a9 (DEFAULT_TABLE table=test-table [id=6f249e3f4e174a2b9c292e06be49d969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:46.581633 1093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd815e00bb504409989fcf136f34f4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:46.583824 1464 tablet_bootstrap.cc:492] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Bootstrap starting.
I20260812 00:15:46.584532 1464 tablet_bootstrap.cc:654] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:46.584391 855 tablet_service.cc:1467] Processing CreateTablet for tablet bd815e00bb504409989fcf136f34f4a9 (DEFAULT_TABLE table=test-table [id=6f249e3f4e174a2b9c292e06be49d969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:46.584544 1467 tablet_bootstrap.cc:492] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Bootstrap starting.
I20260812 00:15:46.584777 855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd815e00bb504409989fcf136f34f4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:46.585124 1464 log.cc:824] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:46.585173 1467 tablet_bootstrap.cc:654] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:46.585777 1467 log.cc:824] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:46.585855 1464 tablet_bootstrap.cc:492] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: No bootstrap required, opened a new log
I20260812 00:15:46.585924 1464 ts_tablet_manager.cc:1397] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:46.586480 1467 tablet_bootstrap.cc:492] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: No bootstrap required, opened a new log
I20260812 00:15:46.586553 1467 ts_tablet_manager.cc:1397] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:46.587472 1464 raft_consensus.cc:348] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.587628 1464 raft_consensus.cc:374] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:46.587670 1464 raft_consensus.cc:729] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3e5271bbad45d7aab2a9f4c14aa604, State: Initialized, Role: FOLLOWER
I20260812 00:15:46.587783 1464 consensus_queue.cc:260] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.588090 1464 ts_tablet_manager.cc:1428] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:46.588228 1474 tablet_bootstrap.cc:492] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: Bootstrap starting.
I20260812 00:15:46.588568 1467 raft_consensus.cc:348] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.588753 1467 raft_consensus.cc:374] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:46.588809 1467 raft_consensus.cc:729] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c1ad24526143c3a2aee26fa4fb24a4, State: Initialized, Role: FOLLOWER
I20260812 00:15:46.588852 1474 tablet_bootstrap.cc:654] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:46.588943 1467 consensus_queue.cc:260] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.589265 1467 ts_tablet_manager.cc:1428] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:46.589274 1209 heartbeater.cc:503] Master 127.31.227.62:45865 was elected leader, sending a full tablet report...
I20260812 00:15:46.589398 1438 heartbeater.cc:503] Master 127.31.227.62:45865 was elected leader, sending a full tablet report...
I20260812 00:15:46.589387 1474 log.cc:824] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:46.590071 1474 tablet_bootstrap.cc:492] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: No bootstrap required, opened a new log
I20260812 00:15:46.590152 1474 ts_tablet_manager.cc:1397] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:46.591506 1474 raft_consensus.cc:348] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.591652 1474 raft_consensus.cc:374] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:46.591701 1474 raft_consensus.cc:729] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc74799e40a4f75a6edb9538652f5b3, State: Initialized, Role: FOLLOWER
I20260812 00:15:46.591805 1474 consensus_queue.cc:260] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.592064 1474 ts_tablet_manager.cc:1428] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:46.592103 969 heartbeater.cc:503] Master 127.31.227.62:45865 was elected leader, sending a full tablet report...
W20260812 00:15:46.625059 1210 tablet.cc:2369] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:46.721268 970 tablet.cc:2369] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:15:46.798538 1439 tablet.cc:2369] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:15:46.803784 1481 raft_consensus.cc:482] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:46.803934 1481 raft_consensus.cc:504] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.804299 1481 leader_election.cc:288] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e3e5271bbad45d7aab2a9f4c14aa604 (127.31.227.3:41835), 84c1ad24526143c3a2aee26fa4fb24a4 (127.31.227.2:40559)
I20260812 00:15:46.807245 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.804572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:51857 (local address 127.31.227.3:41835)
0812 00:15:46.804635 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.804639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.804691 (+ 52us) server_negotiation.cc:407] Connection header received
0812 00:15:46.804777 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.804779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.804790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.804843 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:46.805035 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.805559 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.806510 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.806884 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.806969 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.806978 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.807129 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.807165 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:15:46.807269 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.804790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:36495 (local address 127.31.227.2:40559)
0812 00:15:46.804868 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.804871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.804881 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:46.804913 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.804914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.804927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.805000 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:46.805175 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.805643 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.806545 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.806912 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.807001 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.807010 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.807152 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.807189 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 00:15:46.807387 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.804474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:41835 (local address 127.31.227.1:51857)
0812 00:15:46.804620 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.804631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.804693 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.804870 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.804872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.804875 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:46.804955 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.804957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.805646 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.805647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.806413 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.806415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.806971 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.806972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.807052 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.807055 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.807122 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.807165 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260812 00:15:46.807744 1356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd815e00bb504409989fcf136f34f4a9" candidate_uuid: "ecc74799e40a4f75a6edb9538652f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" is_pre_election: true
I20260812 00:15:46.807886 1356 raft_consensus.cc:2454] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc74799e40a4f75a6edb9538652f5b3 in term 0.
I20260812 00:15:46.808089 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.804680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:40559 (local address 127.31.227.1:36495)
0812 00:15:46.804797 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.804805 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:46.804863 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:46.805004 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:46.805005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:46.805007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:46.805098 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.805099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.805730 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.805731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.806453 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:46.806454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.807001 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.807002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:46.807076 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.807079 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:46.807144 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:15:46.807183 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260812 00:15:46.808146 759 leader_election.cc:302] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e3e5271bbad45d7aab2a9f4c14aa604, ecc74799e40a4f75a6edb9538652f5b3; no voters:
I20260812 00:15:46.808352 1481 raft_consensus.cc:2781] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:46.808437 1481 raft_consensus.cc:482] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:46.808477 1481 raft_consensus.cc:3037] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:46.808568 1123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd815e00bb504409989fcf136f34f4a9" candidate_uuid: "ecc74799e40a4f75a6edb9538652f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" is_pre_election: true
I20260812 00:15:46.808718 1123 raft_consensus.cc:2454] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc74799e40a4f75a6edb9538652f5b3 in term 0.
I20260812 00:15:46.809827 1481 raft_consensus.cc:504] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.809962 1481 leader_election.cc:288] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9e3e5271bbad45d7aab2a9f4c14aa604 (127.31.227.3:41835), 84c1ad24526143c3a2aee26fa4fb24a4 (127.31.227.2:40559)
I20260812 00:15:46.810220 1356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd815e00bb504409989fcf136f34f4a9" candidate_uuid: "ecc74799e40a4f75a6edb9538652f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604"
I20260812 00:15:46.810246 1123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd815e00bb504409989fcf136f34f4a9" candidate_uuid: "ecc74799e40a4f75a6edb9538652f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c1ad24526143c3a2aee26fa4fb24a4"
I20260812 00:15:46.810310 1356 raft_consensus.cc:3037] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:46.810343 1123 raft_consensus.cc:3037] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:46.811123 1356 raft_consensus.cc:2454] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc74799e40a4f75a6edb9538652f5b3 in term 1.
I20260812 00:15:46.811192 1123 raft_consensus.cc:2454] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc74799e40a4f75a6edb9538652f5b3 in term 1.
I20260812 00:15:46.811336 759 leader_election.cc:302] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e3e5271bbad45d7aab2a9f4c14aa604, ecc74799e40a4f75a6edb9538652f5b3; no voters:
I20260812 00:15:46.811463 1481 raft_consensus.cc:2781] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:46.811643 1481 raft_consensus.cc:686] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [term 1 LEADER]: Becoming Leader. State: Replica: ecc74799e40a4f75a6edb9538652f5b3, State: Running, Role: LEADER
I20260812 00:15:46.811755 1481 consensus_queue.cc:237] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } }
I20260812 00:15:46.812608 628 catalog_manager.cc:5495] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc74799e40a4f75a6edb9538652f5b3 (127.31.227.1). New cstate: current_term: 1 leader_uuid: "ecc74799e40a4f75a6edb9538652f5b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc74799e40a4f75a6edb9538652f5b3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 44519 } health_report { overall_health: HEALTHY } } }
I20260812 00:15:46.861207 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.858738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.31.227.1:44519)
0812 00:15:46.858805 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:46.858807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:46.858817 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:46.858865 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:46.858867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:46.858896 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:46.858987 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:46.859143 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.859633 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.860406 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:46.860612 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:46.860687 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:46.860856 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:46.860972 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:46.861073 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:46.861098 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 00:15:46.861119 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:15:46.863353 1123 raft_consensus.cc:1261] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 1 FOLLOWER]: Refusing update from remote peer ecc74799e40a4f75a6edb9538652f5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:46.863358 1356 raft_consensus.cc:1261] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 1 FOLLOWER]: Refusing update from remote peer ecc74799e40a4f75a6edb9538652f5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:46.863739 1485 consensus_queue.cc:1035] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c1ad24526143c3a2aee26fa4fb24a4" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 40559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:46.863886 1481 consensus_queue.cc:1035] T bd815e00bb504409989fcf136f34f4a9 P ecc74799e40a4f75a6edb9538652f5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3e5271bbad45d7aab2a9f4c14aa604" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:46.866508 1493 mvcc.cc:205] Tried to move back new op lower bound from 7317478387149107200 to 7317478386941480960. Current Snapshot: MvccSnapshot[applied={T|T < 7317478387149107200}]
I20260812 00:15:46.868541 628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:46.868646 628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:46.869386 628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05c5d9f3d2e6495dabdeb38c50e701f0: Sending DeleteTablet for 3 replicas of tablet bd815e00bb504409989fcf136f34f4a9
I20260812 00:15:46.869740 1322 tablet_service.cc:1514] Processing DeleteTablet for tablet bd815e00bb504409989fcf136f34f4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:46 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260812 00:15:46.869796 855 tablet_service.cc:1514] Processing DeleteTablet for tablet bd815e00bb504409989fcf136f34f4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:46 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:56740
I20260812 00:15:46.869989 1093 tablet_service.cc:1514] Processing DeleteTablet for tablet bd815e00bb504409989fcf136f34f4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:46 UTC) from {username='kudu', principal='kudu/127.31.227.62@KRBTEST.COM'} at 127.0.0.1:48154
I20260812 00:15:46.870042 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 731
I20260812 00:15:46.874835 1509 tablet_replica.cc:330] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: stopping tablet replica
I20260812 00:15:46.875008 1509 raft_consensus.cc:2229] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:46.875134 1509 raft_consensus.cc:2258] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:46.880059 1511 tablet_replica.cc:330] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: stopping tablet replica
I20260812 00:15:46.880220 1511 raft_consensus.cc:2229] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:46.880364 1511 raft_consensus.cc:2258] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:46.881590 1511 ts_tablet_manager.cc:1905] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:46.883371 1511 ts_tablet_manager.cc:1918] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:46.883497 1511 log.cc:1196] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/bd815e00bb504409989fcf136f34f4a9
I20260812 00:15:46.883751 1511 ts_tablet_manager.cc:1939] T bd815e00bb504409989fcf136f34f4a9 P 84c1ad24526143c3a2aee26fa4fb24a4: Deleting consensus metadata
I20260812 00:15:46.884220 598 catalog_manager.cc:4841] TS 84c1ad24526143c3a2aee26fa4fb24a4 (127.31.227.2:40559): tablet bd815e00bb504409989fcf136f34f4a9 (table test-table [id=6f249e3f4e174a2b9c292e06be49d969]) successfully deleted
I20260812 00:15:46.886318 1509 ts_tablet_manager.cc:1905] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:46.888665 1509 ts_tablet_manager.cc:1918] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:46.888826 1509 log.cc:1196] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/bd815e00bb504409989fcf136f34f4a9
I20260812 00:15:46.889117 1509 ts_tablet_manager.cc:1939] T bd815e00bb504409989fcf136f34f4a9 P 9e3e5271bbad45d7aab2a9f4c14aa604: Deleting consensus metadata
I20260812 00:15:46.889585 598 catalog_manager.cc:4841] TS 9e3e5271bbad45d7aab2a9f4c14aa604 (127.31.227.3:41835): tablet bd815e00bb504409989fcf136f34f4a9 (table test-table [id=6f249e3f4e174a2b9c292e06be49d969]) successfully deleted
W20260812 00:15:46.893479 598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:44519 (error 108)
I20260812 00:15:46.893760 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 981
I20260812 00:15:46.895810 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:46.895647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:44519 (local address 127.0.0.1:56762)
0812 00:15:46.895715 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:46.895763 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 00:15:46.895963 598 catalog_manager.cc:4584] TS ecc74799e40a4f75a6edb9538652f5b3 (127.31.227.1:44519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd815e00bb504409989fcf136f34f4a9: Network error: Client connection negotiation failed: client connection to 127.31.227.1:44519: connect: Connection refused (error 111)
I20260812 00:15:46.906255 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1216
I20260812 00:15:46.919873 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 560
2026-08-12T00:15:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T00:15:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:46Z Disabled control of system clock
I20260812 00:15:46.953717 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40071
--webserver_interface=127.31.227.62
--webserver_port=0
--builtin_ntp_servers=127.31.227.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:40071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:47.036098 1517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:47.036298 1517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:47.036319 1517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:47.037329 1517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:47.037359 1517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:47.037372 1517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:47.037384 1517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:47.037396 1517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:47.038714 1517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:40071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:47.038956 1517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:47.039291 1517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:47.044658 1525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:47.044809 1527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:47.044894 1517 server_base.cc:1034] running on GCE node
W20260812 00:15:47.044654 1523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:47.045374 1517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:47.045779 1517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:47.046974 1517 hybrid_clock.cc:648] HybridClock initialized: now 1786493747046927 us; error 53 us; skew 500 ppm
I20260812 00:15:47.048573 1517 webserver.cc:460] Webserver started at http://127.31.227.62:35659/ using document root <none> and password file <none>
I20260812 00:15:47.048854 1517 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:47.048924 1517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:47.049075 1517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:47.050311 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "828149bd8cb704bfb3ed949a7e00b347"
server_key_iv: "8c8dfefc3f0c4ddc5057a320e2de4ae1"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.050617 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "828149bd8cb704bfb3ed949a7e00b347"
server_key_iv: "8c8dfefc3f0c4ddc5057a320e2de4ae1"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.051841 1517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:47.052563 1542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.052752 1517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:47.052851 1517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "828149bd8cb704bfb3ed949a7e00b347"
server_key_iv: "8c8dfefc3f0c4ddc5057a320e2de4ae1"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.052939 1517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:47.060173 1517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:47.060443 1517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:47.060585 1517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:47.064477 1517 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:40071
I20260812 00:15:47.064525 1634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:40071 every 8 connection(s)
I20260812 00:15:47.064839 1517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:47.065768 1636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:47.067850 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69: Bootstrap starting.
I20260812 00:15:47.068254 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1517
I20260812 00:15:47.068389 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:47.068501 1636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:47.068614 32652 external_mini_cluster.cc:1428] Setting key a8ab6397a69d2e9599c7beb0542a996d
I20260812 00:15:47.068853 1636 log.cc:824] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:47.069999 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69: No bootstrap required, opened a new log
I20260812 00:15:47.071766 1636 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40071 } }
I20260812 00:15:47.071892 1636 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:47.071915 1636 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aee8c4ac19434ea5ff3d1799ef1f69, State: Initialized, Role: FOLLOWER
I20260812 00:15:47.072047 1636 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40071 } }
I20260812 00:15:47.072120 1636 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:47.072142 1636 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:47.072213 1636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:47.072755 1636 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40071 } }
I20260812 00:15:47.072876 1636 leader_election.cc:302] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4aee8c4ac19434ea5ff3d1799ef1f69; no voters:
I20260812 00:15:47.073050 1636 leader_election.cc:288] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:47.073083 1647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:47.073184 1647 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [term 1 LEADER]: Becoming Leader. State: Replica: a4aee8c4ac19434ea5ff3d1799ef1f69, State: Running, Role: LEADER
I20260812 00:15:47.073264 1647 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40071 } }
I20260812 00:15:47.073315 1636 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:47.073593 1651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4aee8c4ac19434ea5ff3d1799ef1f69. Latest consensus state: current_term: 1 leader_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40071 } } }
I20260812 00:15:47.073691 1651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:47.073873 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aee8c4ac19434ea5ff3d1799ef1f69" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40071 } } }
I20260812 00:15:47.073922 1654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:47.074247 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.069259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.31.227.62:40071)
0812 00:15:47.069465 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.069472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.069495 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:15:47.069933 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.069939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.069993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.070144 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:47.070439 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.071292 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.072245 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.072652 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.072781 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.073557 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:47.073562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:47.073562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:47.073762 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:47.073807 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:47.073821 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:47.073866 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.074011 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.074093 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260812 00:15:47.074534 1654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:47.073947 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:47.076215 1654 catalog_manager.cc:1347] Generated new cluster ID: 1dc5b8507b994cb9803c3a7fbd0bdf0b
I20260812 00:15:47.076265 1654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:47.094410 1654 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:47.094971 1654 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:47.102345 1654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69: Generated new TSK 0
I20260812 00:15:47.102548 1654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:47.105711 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.1:0
--local_ip_for_outbound_sockets=127.31.227.1
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40071
--builtin_ntp_servers=127.31.227.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:47.193557 1681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:47.193747 1681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:47.193768 1681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:47.194898 1681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:47.194937 1681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:47.194962 1681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.1
I20260812 00:15:47.196285 1681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.227.1
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.1
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:47.196514 1681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:47.196832 1681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:47.202798 1688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:47.202781 1687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:47.202931 1693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:47.203223 1681 server_base.cc:1034] running on GCE node
I20260812 00:15:47.203459 1681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:47.203729 1681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:47.204911 1681 hybrid_clock.cc:648] HybridClock initialized: now 1786493747204896 us; error 27 us; skew 500 ppm
I20260812 00:15:47.206331 1681 webserver.cc:460] Webserver started at http://127.31.227.1:33883/ using document root <none> and password file <none>
I20260812 00:15:47.206559 1681 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:47.206601 1681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:47.206694 1681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:47.207726 1681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data/instance:
uuid: "c57c360a9a8d4e77a73bc7a7b97927c3"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "e5e1ed7049cffd4e5e05c194075990b3"
server_key_iv: "71456279859d2447cba0e1eaea89d87b"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.208033 1681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance:
uuid: "c57c360a9a8d4e77a73bc7a7b97927c3"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "e5e1ed7049cffd4e5e05c194075990b3"
server_key_iv: "71456279859d2447cba0e1eaea89d87b"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.209292 1681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:47.210058 1703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.210235 1681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.210330 1681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal
uuid: "c57c360a9a8d4e77a73bc7a7b97927c3"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "e5e1ed7049cffd4e5e05c194075990b3"
server_key_iv: "71456279859d2447cba0e1eaea89d87b"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.210430 1681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:47.229049 1681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:47.229410 1681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:47.229661 1681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:47.229934 1681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:47.230276 1681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:47.230345 1681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.230391 1681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:47.230422 1681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.236634 1681 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.1:38137
I20260812 00:15:47.236673 1902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.1:38137 every 8 connection(s)
I20260812 00:15:47.237017 1681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/data/info.pb
I20260812 00:15:47.242353 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.237790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.1:59315 (local address 127.31.227.62:40071)
0812 00:15:47.237887 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.237889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.238085 (+ 196us) server_negotiation.cc:407] Connection header received
0812 00:15:47.238581 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.238583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.238595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.238661 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:47.239304 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.239799 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.240798 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.241043 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.241101 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.242071 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:47.242073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:47.242073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:47.242078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:47.242092 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:47.242095 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:47.242174 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.242256 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.242279 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 00:15:47.242693 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1681
I20260812 00:15:47.242811 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-0/wal/instance
I20260812 00:15:47.243049 32652 external_mini_cluster.cc:1428] Setting key cfcbc75a63e5d764742febbe2d73ba99
I20260812 00:15:47.243409 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.237755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40071 (local address 127.31.227.1:59315)
0812 00:15:47.237950 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.237983 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.238467 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.238715 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.238718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.238726 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:47.239229 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.239233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.239885 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.239886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.240700 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.240704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.241149 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.241150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.241301 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.241910 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:47.241911 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:47.241947 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:47.241976 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:47.242170 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:47.242175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:47.242178 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.242245 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.242366 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260812 00:15:47.244179 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.2:0
--local_ip_for_outbound_sockets=127.31.227.2
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40071
--builtin_ntp_servers=127.31.227.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 00:15:47.244277 1904 heartbeater.cc:347] Connected to a master server at 127.31.227.62:40071
I20260812 00:15:47.244391 1904 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:47.244580 1904 heartbeater.cc:511] Master 127.31.227.62:40071 requested a full tablet report, sending...
I20260812 00:15:47.245221 1574 ts_manager.cc:194] Registered new tserver with Master: c57c360a9a8d4e77a73bc7a7b97927c3 (127.31.227.1:38137)
I20260812 00:15:47.246126 1574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.1:59315
W20260812 00:15:47.327018 1908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:47.327201 1908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:47.327222 1908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:47.328337 1908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:47.328384 1908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:47.328434 1908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.2
I20260812 00:15:47.329825 1908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.227.2
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.2
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:47.330052 1908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:47.330349 1908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:47.335381 1924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:47.335434 1917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:47.335453 1922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:47.335546 1908 server_base.cc:1034] running on GCE node
I20260812 00:15:47.335752 1908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:47.335985 1908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:47.337160 1908 hybrid_clock.cc:648] HybridClock initialized: now 1786493747337149 us; error 28 us; skew 500 ppm
I20260812 00:15:47.338405 1908 webserver.cc:460] Webserver started at http://127.31.227.2:33555/ using document root <none> and password file <none>
I20260812 00:15:47.338626 1908 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:47.338693 1908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:47.338850 1908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:47.339928 1908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data/instance:
uuid: "be4d477f33d3466f8e9dc689f64695f2"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "25aef6867d0484ee949b9ba92ba51d30"
server_key_iv: "e39fa6f7d36dc14ccb8ea822071152b5"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.340283 1908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance:
uuid: "be4d477f33d3466f8e9dc689f64695f2"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "25aef6867d0484ee949b9ba92ba51d30"
server_key_iv: "e39fa6f7d36dc14ccb8ea822071152b5"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.341625 1908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:47.342425 1939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.342589 1908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:47.342689 1908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal
uuid: "be4d477f33d3466f8e9dc689f64695f2"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "25aef6867d0484ee949b9ba92ba51d30"
server_key_iv: "e39fa6f7d36dc14ccb8ea822071152b5"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.342775 1908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:47.373260 1908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:47.373693 1908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:47.373899 1908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:47.374189 1908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:47.374589 1908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:47.374639 1908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.374682 1908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:47.374709 1908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.380771 1908 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.2:35541
I20260812 00:15:47.380822 2114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.2:35541 every 8 connection(s)
I20260812 00:15:47.381156 1908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/data/info.pb
I20260812 00:15:47.386719 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.381952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.2:60597 (local address 127.31.227.62:40071)
0812 00:15:47.382000 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.382002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.382297 (+ 295us) server_negotiation.cc:407] Connection header received
0812 00:15:47.382851 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.382852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.382866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.382921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:47.383650 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.384154 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.385164 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.385364 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.385442 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.386402 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:47.386403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:47.386403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:47.386409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:47.386431 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:47.386435 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:47.386507 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.386607 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.386639 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:15:47.386940 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.381927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40071 (local address 127.31.227.2:60597)
0812 00:15:47.382159 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.382194 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.382742 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.382996 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.383000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.383016 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:47.383543 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.383547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.384255 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.384256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.385054 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.385056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.385454 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.385455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.385645 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.386194 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:47.386194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:47.386222 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:47.386244 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:47.386512 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:47.386519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:47.386523 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.386609 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.386712 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260812 00:15:47.387476 2115 heartbeater.cc:347] Connected to a master server at 127.31.227.62:40071
I20260812 00:15:47.387575 2115 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:47.387804 2115 heartbeater.cc:511] Master 127.31.227.62:40071 requested a full tablet report, sending...
I20260812 00:15:47.388217 1574 ts_manager.cc:194] Registered new tserver with Master: be4d477f33d3466f8e9dc689f64695f2 (127.31.227.2:35541)
I20260812 00:15:47.388576 1574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.2:60597
I20260812 00:15:47.390467 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 1908
I20260812 00:15:47.390578 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal/instance
I20260812 00:15:47.390803 32652 external_mini_cluster.cc:1428] Setting key 0f84dcac572eaec4beb1b183018f371a
I20260812 00:15:47.391767 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.3:0
--local_ip_for_outbound_sockets=127.31.227.3
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40071
--builtin_ntp_servers=127.31.227.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:15:47.476212 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:47.476418 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:47.476438 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:47.477617 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:47.477658 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:47.477681 2120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.3
I20260812 00:15:47.479054 2120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.227.3
--webserver_port=0
--tserver_master_addrs=127.31.227.62:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.3
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:47.479297 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:47.479602 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:47.484679 2128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:47.484809 2120 server_base.cc:1034] running on GCE node
W20260812 00:15:47.484656 2131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:47.484882 2129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:47.485114 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:47.485414 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:47.486577 2120 hybrid_clock.cc:648] HybridClock initialized: now 1786493747486567 us; error 39 us; skew 500 ppm
I20260812 00:15:47.487998 2120 webserver.cc:460] Webserver started at http://127.31.227.3:44681/ using document root <none> and password file <none>
I20260812 00:15:47.488243 2120 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:47.488325 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:47.488466 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:47.489620 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data/instance:
uuid: "345d09a598ff4ca287587764458e5f0f"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "1f82e786c37947e9af1cba4ad958a270"
server_key_iv: "6809266dcfad082017cf2cf36e8c3f82"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.489986 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance:
uuid: "345d09a598ff4ca287587764458e5f0f"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "1f82e786c37947e9af1cba4ad958a270"
server_key_iv: "6809266dcfad082017cf2cf36e8c3f82"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.491277 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:15:47.491990 2143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.492164 2120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:15:47.492266 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal
uuid: "345d09a598ff4ca287587764458e5f0f"
format_stamp: "Formatted at 2026-08-12 00:15:47 on dist-test-slave-zr1t"
server_key: "1f82e786c37947e9af1cba4ad958a270"
server_key_iv: "6809266dcfad082017cf2cf36e8c3f82"
server_key_version: "encryptionkey@0"
I20260812 00:15:47.492367 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:47.502238 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:47.502559 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:47.502709 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:47.502936 2120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:15:47.503273 2120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:15:47.503335 2120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.503379 2120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:15:47.503403 2120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:47.509425 2120 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.3:33637
I20260812 00:15:47.509497 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.3:33637 every 8 connection(s)
I20260812 00:15:47.509851 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/data/info.pb
I20260812 00:15:47.514910 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.510604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:40011 (local address 127.31.227.62:40071)
0812 00:15:47.510668 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.510671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.510914 (+ 243us) server_negotiation.cc:407] Connection header received
0812 00:15:47.511422 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.511423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.511433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.511496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:47.512165 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.512651 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.513680 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.513857 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.513911 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.514699 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:47.514700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:47.514700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:47.514704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:47.514713 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:47.514716 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:47.514754 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.514836 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.514855 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:15:47.515139 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.510596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:40071 (local address 127.31.227.3:40011)
0812 00:15:47.510794 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.510833 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.511334 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.511577 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.511580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.511588 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:15:47.512096 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.512101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.512725 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.512725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.513609 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.513612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.513930 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.513931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.514036 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.514578 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:15:47.514579 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:15:47.514608 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:15:47.514630 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:15:47.514769 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:15:47.514774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:15:47.514776 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.514831 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.514908 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260812 00:15:47.515694 2352 heartbeater.cc:347] Connected to a master server at 127.31.227.62:40071
I20260812 00:15:47.515800 2352 heartbeater.cc:464] Registering TS with master...
I20260812 00:15:47.515981 2352 heartbeater.cc:511] Master 127.31.227.62:40071 requested a full tablet report, sending...
I20260812 00:15:47.516335 1574 ts_manager.cc:194] Registered new tserver with Master: 345d09a598ff4ca287587764458e5f0f (127.31.227.3:33637)
I20260812 00:15:47.516654 1574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.3:40011
I20260812 00:15:47.517690 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2120
I20260812 00:15:47.517799 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal/instance
I20260812 00:15:47.518002 32652 external_mini_cluster.cc:1428] Setting key 35a8cdace9536dc385369060f372885a
I20260812 00:15:47.518941 32652 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:15:47.523352 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.520399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.31.227.62:40071)
0812 00:15:47.520468 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.520470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.520586 (+ 116us) server_negotiation.cc:407] Connection header received
0812 00:15:47.520665 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.520666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.520677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.520741 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:47.520915 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.521389 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.522164 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.522360 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.522434 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.523096 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:47.523098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:47.523098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:47.523102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:47.523114 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:47.523118 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:47.523185 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.523247 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.523267 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:15:47.524948 1574 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:15:47.526554 1574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:15:47.532791 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.528636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.3:33637 (local address 127.0.0.1:46168)
0812 00:15:47.528957 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.528968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.529006 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.529376 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.529377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.529380 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:47.529429 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.529430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.530720 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.530721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.531761 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.531763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.532515 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.532590 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.532602 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.532673 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.532725 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260812 00:15:47.532800 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.529001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.31.227.3:33637)
0812 00:15:47.529155 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.529160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.529171 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:15:47.529209 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.529211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.529227 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.529310 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:47.529510 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.530634 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.531851 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.532424 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532514 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.532599 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.532686 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.532721 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260812 00:15:47.533108 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.528745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.31.227.2:35541)
0812 00:15:47.529110 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.529114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.529122 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:15:47.529153 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.529155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.529167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.529492 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:47.529574 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.530375 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.531796 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.532492 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532557 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.532638 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.532974 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.533020 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":112,"threads_started":1}
I20260812 00:15:47.532907 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.528635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:35541 (local address 127.0.0.1:50608)
0812 00:15:47.528787 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.528828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.528893 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.529282 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.529285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.529287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:47.529366 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.529368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.530456 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.530457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.531692 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.531695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.532675 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.532736 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.532741 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.532794 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.532837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260812 00:15:47.533154 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.528859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:38137 (local address 127.0.0.1:40384)
0812 00:15:47.529369 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.529379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.529423 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.529753 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.529754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.529756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:47.529791 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.529792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.530967 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.530968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.531977 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.531980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.532944 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.532999 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.533002 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.533051 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.533084 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":79,"threads_started":1}
I20260812 00:15:47.533691 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.529174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.31.227.1:38137)
0812 00:15:47.529592 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.529596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.529606 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:47.529644 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.529645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.529659 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.529739 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:47.529871 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.530876 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532064 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.532763 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.532833 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.532908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.533241 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.533613 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":89,"threads_started":1}
I20260812 00:15:47.533751 2224 tablet_service.cc:1467] Processing CreateTablet for tablet eb83cd2d85f442ab916070dd39450e7b (DEFAULT_TABLE table=test-table [id=8ce22e6bdd494d0da61c23fbb693e27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:47.533885 2013 tablet_service.cc:1467] Processing CreateTablet for tablet eb83cd2d85f442ab916070dd39450e7b (DEFAULT_TABLE table=test-table [id=8ce22e6bdd494d0da61c23fbb693e27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:47.534142 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb83cd2d85f442ab916070dd39450e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:47.534205 2013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb83cd2d85f442ab916070dd39450e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:47.536623 2369 tablet_bootstrap.cc:492] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Bootstrap starting.
I20260812 00:15:47.536639 2370 tablet_bootstrap.cc:492] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Bootstrap starting.
I20260812 00:15:47.537317 2369 tablet_bootstrap.cc:654] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:47.537424 2370 tablet_bootstrap.cc:654] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:47.537359 1796 tablet_service.cc:1467] Processing CreateTablet for tablet eb83cd2d85f442ab916070dd39450e7b (DEFAULT_TABLE table=test-table [id=8ce22e6bdd494d0da61c23fbb693e27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:15:47.537698 2369 log.cc:824] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:47.537755 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb83cd2d85f442ab916070dd39450e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:47.537822 2370 log.cc:824] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:47.538542 2369 tablet_bootstrap.cc:492] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: No bootstrap required, opened a new log
I20260812 00:15:47.538597 2370 tablet_bootstrap.cc:492] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: No bootstrap required, opened a new log
I20260812 00:15:47.538617 2369 ts_tablet_manager.cc:1397] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 00:15:47.538646 2370 ts_tablet_manager.cc:1397] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:47.540282 2374 tablet_bootstrap.cc:492] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3: Bootstrap starting.
I20260812 00:15:47.540900 2374 tablet_bootstrap.cc:654] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:47.541208 2374 log.cc:824] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:47.541361 2369 raft_consensus.cc:348] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.541566 2369 raft_consensus.cc:374] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:47.541623 2369 raft_consensus.cc:729] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4d477f33d3466f8e9dc689f64695f2, State: Initialized, Role: FOLLOWER
I20260812 00:15:47.541586 2370 raft_consensus.cc:348] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.541734 2370 raft_consensus.cc:374] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:47.541781 2370 raft_consensus.cc:729] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345d09a598ff4ca287587764458e5f0f, State: Initialized, Role: FOLLOWER
I20260812 00:15:47.541743 2369 consensus_queue.cc:260] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.541920 2370 consensus_queue.cc:260] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.542039 2369 ts_tablet_manager.cc:1428] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260812 00:15:47.542110 2115 heartbeater.cc:503] Master 127.31.227.62:40071 was elected leader, sending a full tablet report...
I20260812 00:15:47.542166 2370 ts_tablet_manager.cc:1428] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:15:47.542279 2352 heartbeater.cc:503] Master 127.31.227.62:40071 was elected leader, sending a full tablet report...
I20260812 00:15:47.542371 2374 tablet_bootstrap.cc:492] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3: No bootstrap required, opened a new log
I20260812 00:15:47.542433 2374 ts_tablet_manager.cc:1397] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:15:47.544411 2374 raft_consensus.cc:348] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.544528 2374 raft_consensus.cc:374] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:47.544548 2374 raft_consensus.cc:729] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c360a9a8d4e77a73bc7a7b97927c3, State: Initialized, Role: FOLLOWER
I20260812 00:15:47.544670 2374 consensus_queue.cc:260] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.544919 2374 ts_tablet_manager.cc:1428] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:15:47.544991 1904 heartbeater.cc:503] Master 127.31.227.62:40071 was elected leader, sending a full tablet report...
I20260812 00:15:47.585417 2379 raft_consensus.cc:482] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:15:47.585574 2379 raft_consensus.cc:504] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.586455 2379 leader_election.cc:288] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be4d477f33d3466f8e9dc689f64695f2 (127.31.227.2:35541), c57c360a9a8d4e77a73bc7a7b97927c3 (127.31.227.1:38137)
I20260812 00:15:47.589995 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.586198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:38137 (local address 127.31.227.3:59509)
0812 00:15:47.586529 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.586552 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.586604 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.586807 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.586809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.586810 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:47.586902 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.586904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.587554 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.587555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.588856 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.588859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.589713 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.589715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.589792 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.589797 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.589851 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.589891 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":185,"thread_start_us":81,"threads_started":1}
I20260812 00:15:47.590008 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.586264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:59509 (local address 127.31.227.1:38137)
0812 00:15:47.586309 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.586311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.586601 (+ 290us) server_negotiation.cc:407] Connection header received
0812 00:15:47.586667 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.586668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.586683 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.586745 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:47.586986 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.587465 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.588945 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.589626 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.589711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.589722 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.589860 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.589915 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:15:47.590155 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.586264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.227.3:39311 (local address 127.31.227.2:35541)
0812 00:15:47.586327 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.586330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.586339 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:15:47.586369 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.586370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.586385 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.586445 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:15:47.586677 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.587508 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.589033 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.589700 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.589766 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.589775 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.590031 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.590071 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:15:47.590309 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.586198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.2:35541 (local address 127.31.227.3:39311)
0812 00:15:47.586237 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.586248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:15:47.586290 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:15:47.586470 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:15:47.586471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:15:47.586473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:15:47.586572 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.586575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.587608 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.587609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.588856 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:15:47.588859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.589806 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.589807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:15:47.589884 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.589886 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:15:47.589930 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 00:15:47.589974 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":52}
I20260812 00:15:47.590811 1828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb83cd2d85f442ab916070dd39450e7b" candidate_uuid: "345d09a598ff4ca287587764458e5f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" is_pre_election: true
I20260812 00:15:47.590803 2042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb83cd2d85f442ab916070dd39450e7b" candidate_uuid: "345d09a598ff4ca287587764458e5f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4d477f33d3466f8e9dc689f64695f2" is_pre_election: true
I20260812 00:15:47.590984 1828 raft_consensus.cc:2454] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345d09a598ff4ca287587764458e5f0f in term 0.
I20260812 00:15:47.590987 2042 raft_consensus.cc:2454] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345d09a598ff4ca287587764458e5f0f in term 0.
I20260812 00:15:47.591223 2152 leader_election.cc:302] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345d09a598ff4ca287587764458e5f0f, be4d477f33d3466f8e9dc689f64695f2; no voters:
I20260812 00:15:47.591359 2379 raft_consensus.cc:2781] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:15:47.591419 2379 raft_consensus.cc:482] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:15:47.591441 2379 raft_consensus.cc:3037] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:47.592123 2379 raft_consensus.cc:504] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.592283 2379 leader_election.cc:288] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [CANDIDATE]: Term 1 election: Requested vote from peers be4d477f33d3466f8e9dc689f64695f2 (127.31.227.2:35541), c57c360a9a8d4e77a73bc7a7b97927c3 (127.31.227.1:38137)
I20260812 00:15:47.592469 1828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb83cd2d85f442ab916070dd39450e7b" candidate_uuid: "345d09a598ff4ca287587764458e5f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3"
I20260812 00:15:47.592460 2042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb83cd2d85f442ab916070dd39450e7b" candidate_uuid: "345d09a598ff4ca287587764458e5f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4d477f33d3466f8e9dc689f64695f2"
I20260812 00:15:47.592536 1828 raft_consensus.cc:3037] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:47.592536 2042 raft_consensus.cc:3037] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:47.593447 2042 raft_consensus.cc:2454] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345d09a598ff4ca287587764458e5f0f in term 1.
I20260812 00:15:47.593454 1828 raft_consensus.cc:2454] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345d09a598ff4ca287587764458e5f0f in term 1.
I20260812 00:15:47.593663 2146 leader_election.cc:302] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345d09a598ff4ca287587764458e5f0f, c57c360a9a8d4e77a73bc7a7b97927c3; no voters:
I20260812 00:15:47.593817 2379 raft_consensus.cc:2781] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:47.594013 2379 raft_consensus.cc:686] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 1 LEADER]: Becoming Leader. State: Replica: 345d09a598ff4ca287587764458e5f0f, State: Running, Role: LEADER
I20260812 00:15:47.594125 2379 consensus_queue.cc:237] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } }
I20260812 00:15:47.595072 1572 catalog_manager.cc:5495] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 345d09a598ff4ca287587764458e5f0f (127.31.227.3). New cstate: current_term: 1 leader_uuid: "345d09a598ff4ca287587764458e5f0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345d09a598ff4ca287587764458e5f0f" member_type: VOTER last_known_addr { host: "127.31.227.3" port: 33637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 } health_report { overall_health: UNKNOWN } } }
I20260812 00:15:47.614599 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.611470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.31.227.3:33637)
0812 00:15:47.611546 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:47.611549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:47.611559 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:15:47.611610 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:47.611612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:47.611636 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:47.611727 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:15:47.611870 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.612590 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.613662 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:47.613930 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:47.614031 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:47.614214 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:15:47.614332 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:15:47.614422 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:47.614461 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:15:47.614479 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:15:47.616451 1828 raft_consensus.cc:1261] T eb83cd2d85f442ab916070dd39450e7b P c57c360a9a8d4e77a73bc7a7b97927c3 [term 1 FOLLOWER]: Refusing update from remote peer 345d09a598ff4ca287587764458e5f0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:47.616451 2042 raft_consensus.cc:1261] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 1 FOLLOWER]: Refusing update from remote peer 345d09a598ff4ca287587764458e5f0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:15:47.616792 2379 consensus_queue.cc:1035] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c360a9a8d4e77a73bc7a7b97927c3" member_type: VOTER last_known_addr { host: "127.31.227.1" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:47.616930 2384 consensus_queue.cc:1035] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "be4d477f33d3466f8e9dc689f64695f2" member_type: VOTER last_known_addr { host: "127.31.227.2" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:15:47.619839 2394 mvcc.cc:205] Tried to move back new op lower bound from 7317478390234439680 to 7317478390146617344. Current Snapshot: MvccSnapshot[applied={T|T < 7317478390234439680}]
I20260812 00:15:47.620064 2398 mvcc.cc:205] Tried to move back new op lower bound from 7317478390234439680 to 7317478390146617344. Current Snapshot: MvccSnapshot[applied={T|T < 7317478390234439680}]
I20260812 00:15:47.621968 1572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:47.622061 1572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:15:47.622845 1572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4aee8c4ac19434ea5ff3d1799ef1f69: Sending DeleteTablet for 3 replicas of tablet eb83cd2d85f442ab916070dd39450e7b
I20260812 00:15:47.623193 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet eb83cd2d85f442ab916070dd39450e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:47 UTC) from {username='slave'} at 127.0.0.1:46168
I20260812 00:15:47.623214 1796 tablet_service.cc:1514] Processing DeleteTablet for tablet eb83cd2d85f442ab916070dd39450e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:47 UTC) from {username='slave'} at 127.0.0.1:40384
I20260812 00:15:47.623210 2013 tablet_service.cc:1514] Processing DeleteTablet for tablet eb83cd2d85f442ab916070dd39450e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:15:47 UTC) from {username='slave'} at 127.0.0.1:50608
I20260812 00:15:47.623437 2411 tablet_replica.cc:330] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: stopping tablet replica
I20260812 00:15:47.623548 2411 raft_consensus.cc:2229] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:15:47.623651 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1681
I20260812 00:15:47.636627 2411 raft_consensus.cc:2258] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:47.636889 2410 tablet_replica.cc:330] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: stopping tablet replica
I20260812 00:15:47.637006 2410 raft_consensus.cc:2229] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:15:47.637145 2410 raft_consensus.cc:2258] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:15:47.637559 2411 ts_tablet_manager.cc:1905] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:47.637591 2410 ts_tablet_manager.cc:1905] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:15:47.639343 2411 ts_tablet_manager.cc:1918] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:47.639430 2411 log.cc:1196] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-1/wal/wals/eb83cd2d85f442ab916070dd39450e7b
I20260812 00:15:47.639701 2411 ts_tablet_manager.cc:1939] T eb83cd2d85f442ab916070dd39450e7b P be4d477f33d3466f8e9dc689f64695f2: Deleting consensus metadata
I20260812 00:15:47.640049 2410 ts_tablet_manager.cc:1918] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:15:47.640189 2410 log.cc:1196] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Deleting WAL directory at /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493496733958-32652-0/minicluster-data/ts-2/wal/wals/eb83cd2d85f442ab916070dd39450e7b
I20260812 00:15:47.640163 1551 catalog_manager.cc:4841] TS be4d477f33d3466f8e9dc689f64695f2 (127.31.227.2:35541): tablet eb83cd2d85f442ab916070dd39450e7b (table test-table [id=8ce22e6bdd494d0da61c23fbb693e27f]) successfully deleted
I20260812 00:15:47.640545 2410 ts_tablet_manager.cc:1939] T eb83cd2d85f442ab916070dd39450e7b P 345d09a598ff4ca287587764458e5f0f: Deleting consensus metadata
I20260812 00:15:47.640982 1550 catalog_manager.cc:4841] TS 345d09a598ff4ca287587764458e5f0f (127.31.227.3:33637): tablet eb83cd2d85f442ab916070dd39450e7b (table test-table [id=8ce22e6bdd494d0da61c23fbb693e27f]) successfully deleted
W20260812 00:15:47.642999 1550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.227.1:38137 (error 108)
I20260812 00:15:47.643566 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:47.643408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.1:38137 (local address 127.0.0.1:40386)
0812 00:15:47.643474 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 00:15:47.643525 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:15:47.643657 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1908
W20260812 00:15:47.650645 1550 catalog_manager.cc:4584] TS c57c360a9a8d4e77a73bc7a7b97927c3 (127.31.227.1:38137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb83cd2d85f442ab916070dd39450e7b: Network error: Client connection negotiation failed: client connection to 127.31.227.1:38137: connect: Connection refused (error 111)
I20260812 00:15:47.659633 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2120
I20260812 00:15:47.674445 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 1517
2026-08-12T00:15:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (753 ms)
[----------] 6 tests from EncryptionPolicyTest (9590 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:47 dist-test-slave-zr1t krb5kdc[2420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:47 dist-test-slave-zr1t krb5kdc[2420](info): set up 2 sockets
Aug 12 00:15:47 dist-test-slave-zr1t krb5kdc[2420](info): commencing operation
krb5kdc: starting...
W20260812 00:15:48.937711 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:49 dist-test-slave-zr1t krb5kdc[2420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:49.057914 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:49.140075 2440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:49.140280 2440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:49.140301 2440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:49.141391 2440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:49.141425 2440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:49.141439 2440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:49.141453 2440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:49.141464 2440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:49.143018 2440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:49.143211 2440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:49.143399 2440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:49.148304 2451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:49.148311 2449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:49.148409 2440 server_base.cc:1034] running on GCE node
W20260812 00:15:49.148311 2448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:49.148797 2440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:49.149080 2440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:49.150219 2440 hybrid_clock.cc:648] HybridClock initialized: now 1786493749150206 us; error 29 us; skew 500 ppm
Aug 12 00:15:49 dist-test-slave-zr1t krb5kdc[2420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:49.157956 2440 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:49.158404 2440 webserver.cc:460] Webserver started at http://127.0.0.1:46221/ using document root <none> and password file <none>
I20260812 00:15:49.158639 2440 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:49.158710 2440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:49.158876 2440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:49.159966 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "228d51c6feb64d5aa5e3848636702c47"
format_stamp: "Formatted at 2026-08-12 00:15:49 on dist-test-slave-zr1t"
server_key: "d7bf2b1ef5791649f0f56cca3452561f"
server_key_iv: "6627a52bc35cde3065e33d1e27e0cc93"
server_key_version: "encryptionkey@0"
I20260812 00:15:49.160342 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "228d51c6feb64d5aa5e3848636702c47"
format_stamp: "Formatted at 2026-08-12 00:15:49 on dist-test-slave-zr1t"
server_key: "d7bf2b1ef5791649f0f56cca3452561f"
server_key_iv: "6627a52bc35cde3065e33d1e27e0cc93"
server_key_version: "encryptionkey@0"
I20260812 00:15:49.161749 2440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:15:49.162528 2468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:49.162707 2440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:49.162812 2440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "228d51c6feb64d5aa5e3848636702c47"
format_stamp: "Formatted at 2026-08-12 00:15:49 on dist-test-slave-zr1t"
server_key: "d7bf2b1ef5791649f0f56cca3452561f"
server_key_iv: "6627a52bc35cde3065e33d1e27e0cc93"
server_key_version: "encryptionkey@0"
I20260812 00:15:49.162896 2440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:49.187870 2440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:49.189870 2440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:49.190040 2440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:49.194304 2440 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36387
I20260812 00:15:49.194350 2561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36387 every 8 connection(s)
I20260812 00:15:49.194742 2440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:49.195631 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:49.197964 2563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47: Bootstrap starting.
I20260812 00:15:49.198586 2563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:49.198881 2563 log.cc:824] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:49.199609 2563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47: No bootstrap required, opened a new log
I20260812 00:15:49.201269 2563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228d51c6feb64d5aa5e3848636702c47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36387 } }
I20260812 00:15:49.201408 2563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:49.201431 2563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228d51c6feb64d5aa5e3848636702c47, State: Initialized, Role: FOLLOWER
I20260812 00:15:49.201644 2563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228d51c6feb64d5aa5e3848636702c47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36387 } }
I20260812 00:15:49.201728 2563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:49.201771 2563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:49.201821 2563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:49.202361 2563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228d51c6feb64d5aa5e3848636702c47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36387 } }
I20260812 00:15:49.202483 2563 leader_election.cc:302] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 228d51c6feb64d5aa5e3848636702c47; no voters:
I20260812 00:15:49.202682 2563 leader_election.cc:288] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:49.202735 2570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:49.202936 2563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:49.203133 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2440
I20260812 00:15:49.203256 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:49.203488 32652 external_mini_cluster.cc:1428] Setting key fd950134df533c63dadf46e01e787c35
I20260812 00:15:49.202907 2570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [term 1 LEADER]: Becoming Leader. State: Replica: 228d51c6feb64d5aa5e3848636702c47, State: Running, Role: LEADER
I20260812 00:15:49.203684 2570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228d51c6feb64d5aa5e3848636702c47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36387 } }
I20260812 00:15:49.204300 2575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 228d51c6feb64d5aa5e3848636702c47. Latest consensus state: current_term: 1 leader_uuid: "228d51c6feb64d5aa5e3848636702c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228d51c6feb64d5aa5e3848636702c47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36387 } } }
I20260812 00:15:49.204335 2573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "228d51c6feb64d5aa5e3848636702c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228d51c6feb64d5aa5e3848636702c47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36387 } } }
I20260812 00:15:49.204394 2575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:49.204427 2573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47 [sys.catalog]: This master's current role is: LEADER
W20260812 00:15:49.205303 2594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:49.205413 2594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:49.205580 2595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:49.205771 2595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:49.207350 2595 catalog_manager.cc:1347] Generated new cluster ID: f8073474d5684ed197d9143d0df111ad
I20260812 00:15:49.207419 2595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:15:49 dist-test-slave-zr1t krb5kdc[2420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:15:49.213275 2595 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:49.214007 2595 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:49.215358 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:49.204074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35789 (local address 127.0.0.1:36387)
0812 00:15:49.204580 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:49.204590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:49.204652 (+ 62us) server_negotiation.cc:407] Connection header received
0812 00:15:49.204963 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:49.204971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:49.205033 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:49.205183 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:49.206078 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:49.207275 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:49.208156 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:49.208595 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:49.208715 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:49.211196 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:49.211200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:49.211200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:49.211213 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:49.213754 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:49.214192 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:49.214195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:49.214196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:49.214279 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:49.214523 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:49.214524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:49.214525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:49.214744 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:49.214929 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:49.215041 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:15:49.215150 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":64,"threads_started":1}
I20260812 00:15:49.216377 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:49.219688 2595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 228d51c6feb64d5aa5e3848636702c47: Generated new TSK 0
I20260812 00:15:49.219844 2595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:49.225483 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:49.217697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40729 (local address 127.0.0.1:36387)
0812 00:15:49.217736 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:49.217738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:49.219409 (+ 1671us) server_negotiation.cc:407] Connection header received
0812 00:15:49.219513 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:49.219514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:49.219525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:49.219596 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:49.220233 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:49.220760 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:49.221480 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:49.221744 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:49.221828 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:49.223014 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:49.223015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:49.223016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:49.223023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:49.224325 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:49.224689 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:49.224690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:49.224691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:49.224746 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:49.224982 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:49.224983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:49.224984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:49.225094 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:49.225205 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:49.225317 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:15:49.225378 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:15:49.227347 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2440
W20260812 00:15:49.240690 2417 connection.cc:535] client connection to 127.0.0.1:36387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36387): Connection reset by peer (error 104)
2026-08-12T00:15:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1555 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:49 dist-test-slave-zr1t krb5kdc[2616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:49 dist-test-slave-zr1t krb5kdc[2616](info): set up 2 sockets
Aug 12 00:15:49 dist-test-slave-zr1t krb5kdc[2616](info): commencing operation
krb5kdc: starting...
W20260812 00:15:50.481741 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:50 dist-test-slave-zr1t krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:50.596045 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:50.677611 2637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:50.677860 2637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:50.677896 2637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:50.679039 2637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:50.679108 2637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:50.679150 2637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:50.679178 2637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:50.679210 2637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:50.680682 2637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:50.680956 2637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:50.681183 2637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:50.685999 2657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:50.686007 2655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:50.686266 2637 server_base.cc:1034] running on GCE node
W20260812 00:15:50.686007 2654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:50.686558 2637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:50.686796 2637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:50.687933 2637 hybrid_clock.cc:648] HybridClock initialized: now 1786493750687911 us; error 35 us; skew 500 ppm
Aug 12 00:15:50 dist-test-slave-zr1t krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:50.690560 2637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:50.691025 2637 webserver.cc:460] Webserver started at http://127.0.0.1:43441/ using document root <none> and password file <none>
I20260812 00:15:50.691238 2637 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:50.691283 2637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:50.691430 2637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:50.692476 2637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "0c679eef720a4a63bb05d1ed72e49d50"
format_stamp: "Formatted at 2026-08-12 00:15:50 on dist-test-slave-zr1t"
server_key: "b01709dc13dc25a9fb37bfa82ed3292e"
server_key_iv: "946a4c50ca11383d992518c09bd98fef"
server_key_version: "encryptionkey@0"
I20260812 00:15:50.692835 2637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "0c679eef720a4a63bb05d1ed72e49d50"
format_stamp: "Formatted at 2026-08-12 00:15:50 on dist-test-slave-zr1t"
server_key: "b01709dc13dc25a9fb37bfa82ed3292e"
server_key_iv: "946a4c50ca11383d992518c09bd98fef"
server_key_version: "encryptionkey@0"
I20260812 00:15:50.694320 2637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:15:50.695103 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:50.695355 2637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:15:50.695433 2637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "0c679eef720a4a63bb05d1ed72e49d50"
format_stamp: "Formatted at 2026-08-12 00:15:50 on dist-test-slave-zr1t"
server_key: "b01709dc13dc25a9fb37bfa82ed3292e"
server_key_iv: "946a4c50ca11383d992518c09bd98fef"
server_key_version: "encryptionkey@0"
I20260812 00:15:50.695521 2637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:50.711266 2637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:50.713635 2637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:50.713788 2637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:50.718060 2637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38627
I20260812 00:15:50.718127 2763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38627 every 8 connection(s)
I20260812 00:15:50.718459 2637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:50.719342 2764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:50.720794 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2637
I20260812 00:15:50.720892 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:50.721108 32652 external_mini_cluster.cc:1428] Setting key 9a3d23f639f60f83d11d958204f90304
I20260812 00:15:50.722239 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50: Bootstrap starting.
I20260812 00:15:50.722993 2764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:50.723330 2764 log.cc:824] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:50.724085 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50: No bootstrap required, opened a new log
I20260812 00:15:50.726334 2764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c679eef720a4a63bb05d1ed72e49d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260812 00:15:50.726503 2764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:50.726542 2764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c679eef720a4a63bb05d1ed72e49d50, State: Initialized, Role: FOLLOWER
I20260812 00:15:50.726677 2764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c679eef720a4a63bb05d1ed72e49d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260812 00:15:50.726771 2764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:50.726806 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:50.726868 2764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:50.727455 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:50.721676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39767 (local address 127.0.0.1:38627)
0812 00:15:50.721875 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:50.721885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:50.721919 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:15:50.722503 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:50.722513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:50.722582 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:50.722750 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:50.722919 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:50.724289 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:50.725066 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:50.725748 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:50.725876 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:50.726541 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:50.726545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:50.726545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:50.726752 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:50.726793 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:50.726809 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:50.726952 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:50.727197 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 00:15:50.727230 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260812 00:15:50.727646 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c679eef720a4a63bb05d1ed72e49d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260812 00:15:50.727768 2764 leader_election.cc:302] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c679eef720a4a63bb05d1ed72e49d50; no voters:
I20260812 00:15:50.727952 2764 leader_election.cc:288] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:50.727985 2775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:50.728148 2775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [term 1 LEADER]: Becoming Leader. State: Replica: 0c679eef720a4a63bb05d1ed72e49d50, State: Running, Role: LEADER
I20260812 00:15:50.728201 2764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:50.728283 2775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c679eef720a4a63bb05d1ed72e49d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260812 00:15:50.728693 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:50.728693 2776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c679eef720a4a63bb05d1ed72e49d50. Latest consensus state: current_term: 1 leader_uuid: "0c679eef720a4a63bb05d1ed72e49d50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c679eef720a4a63bb05d1ed72e49d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } } }
I20260812 00:15:50.728787 2776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:50.729707 2787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:50.730258 2787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:50.730513 2778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c679eef720a4a63bb05d1ed72e49d50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c679eef720a4a63bb05d1ed72e49d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } } }
I20260812 00:15:50.730597 2778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:50.732028 2787 catalog_manager.cc:1347] Generated new cluster ID: bdc84f51d92648f8b9b4271feac7d2d4
I20260812 00:15:50.732079 2787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:50.734602 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:50.730324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33021 (local address 127.0.0.1:38627)
0812 00:15:50.730372 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:50.730374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:50.731242 (+ 868us) server_negotiation.cc:407] Connection header received
0812 00:15:50.731289 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:50.731290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:50.731302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:50.731370 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:50.731524 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:50.732217 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:50.733254 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:50.733583 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:50.733641 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:50.734340 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:50.734341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:50.734341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:50.734346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:50.734360 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:50.734364 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:50.734418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:50.734524 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 00:15:50.734534 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:15:50.741415 2787 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:50.741935 2787 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:50.744928 2787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c679eef720a4a63bb05d1ed72e49d50: Generated new TSK 0
I20260812 00:15:50.745100 2787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:50.752547 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2637
W20260812 00:15:50.767673 2614 connection.cc:535] client connection to 127.0.0.1:38627 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38627): Connection reset by peer (error 104)
2026-08-12T00:15:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:50 dist-test-slave-zr1t krb5kdc[2828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:50 dist-test-slave-zr1t krb5kdc[2828](info): set up 2 sockets
Aug 12 00:15:50 dist-test-slave-zr1t krb5kdc[2828](info): commencing operation
krb5kdc: starting...
W20260812 00:15:52.023965 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:52 dist-test-slave-zr1t krb5kdc[2828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:52.142495 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:52.224743 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:52.224998 2846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:52.225036 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:52.226229 2846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:52.226302 2846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:52.226327 2846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:52.226346 2846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:52.226363 2846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:52.227861 2846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:52.228154 2846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:52.228396 2846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:52.233497 2858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:52.233477 2853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:52.233498 2854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:52.233726 2846 server_base.cc:1034] running on GCE node
I20260812 00:15:52.233973 2846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:52.234256 2846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:52.235397 2846 hybrid_clock.cc:648] HybridClock initialized: now 1786493752235400 us; error 59 us; skew 500 ppm
Aug 12 00:15:52 dist-test-slave-zr1t krb5kdc[2828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:52.237872 2846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:52.238278 2846 webserver.cc:460] Webserver started at http://127.0.0.1:45939/ using document root <none> and password file <none>
I20260812 00:15:52.238523 2846 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:52.238569 2846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:52.238732 2846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:52.239738 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "4543aea35bf54e14a4b385a1e7f9c90b"
format_stamp: "Formatted at 2026-08-12 00:15:52 on dist-test-slave-zr1t"
server_key: "62ae956f4ca1294a937c9bb057b782e1"
server_key_iv: "6dde64b4a40b82bffe3dc546ff95c1f9"
server_key_version: "encryptionkey@0"
I20260812 00:15:52.240085 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "4543aea35bf54e14a4b385a1e7f9c90b"
format_stamp: "Formatted at 2026-08-12 00:15:52 on dist-test-slave-zr1t"
server_key: "62ae956f4ca1294a937c9bb057b782e1"
server_key_iv: "6dde64b4a40b82bffe3dc546ff95c1f9"
server_key_version: "encryptionkey@0"
I20260812 00:15:52.241339 2846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:15:52.242142 2868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:52.242329 2846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:52.242420 2846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "4543aea35bf54e14a4b385a1e7f9c90b"
format_stamp: "Formatted at 2026-08-12 00:15:52 on dist-test-slave-zr1t"
server_key: "62ae956f4ca1294a937c9bb057b782e1"
server_key_iv: "6dde64b4a40b82bffe3dc546ff95c1f9"
server_key_version: "encryptionkey@0"
I20260812 00:15:52.242503 2846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:52.245644 2846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:52.247301 2846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:52.247432 2846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:52.251477 2846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38675
I20260812 00:15:52.251549 2949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38675 every 8 connection(s)
I20260812 00:15:52.251837 2846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:52.252689 2950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:52.254997 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b: Bootstrap starting.
I20260812 00:15:52.255515 2950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:52.255792 2950 log.cc:824] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:52.256413 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b: No bootstrap required, opened a new log
I20260812 00:15:52.257484 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 2846
I20260812 00:15:52.257664 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:52.257843 32652 external_mini_cluster.cc:1428] Setting key 4884bf45668b0360b956b19a7d9da8cb
I20260812 00:15:52.258204 2950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260812 00:15:52.258373 2950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:52.258425 2950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4543aea35bf54e14a4b385a1e7f9c90b, State: Initialized, Role: FOLLOWER
I20260812 00:15:52.258582 2950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260812 00:15:52.258646 2950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:52.258667 2950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:52.258728 2950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:52.259442 2950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260812 00:15:52.259555 2950 leader_election.cc:302] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4543aea35bf54e14a4b385a1e7f9c90b; no voters:
I20260812 00:15:52.259716 2950 leader_election.cc:288] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:52.259760 2956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:52.259938 2956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [term 1 LEADER]: Becoming Leader. State: Replica: 4543aea35bf54e14a4b385a1e7f9c90b, State: Running, Role: LEADER
I20260812 00:15:52.260010 2950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:52.260088 2956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260812 00:15:52.260138 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:52.258319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53543 (local address 127.0.0.1:38675)
0812 00:15:52.258507 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:52.258515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:52.258542 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:15:52.258645 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:52.258652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:52.258692 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:52.258845 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:52.259581 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:52.259585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:52.259585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:52.259589 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:52.259630 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:52.259645 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:52.259740 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:52.259826 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 00:15:52.259876 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260812 00:15:52.260705 2958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } } }
I20260812 00:15:52.260823 2958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:52.260704 2959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4543aea35bf54e14a4b385a1e7f9c90b. Latest consensus state: current_term: 1 leader_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4543aea35bf54e14a4b385a1e7f9c90b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } } }
I20260812 00:15:52.261036 2959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:52.261190 2971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:52.261245 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:52.261807 2971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:52.263883 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:52.262811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58539 (local address 127.0.0.1:38675)
0812 00:15:52.262845 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:52.262846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:52.263039 (+ 193us) server_negotiation.cc:407] Connection header received
0812 00:15:52.263106 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:52.263107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:52.263112 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:52.263165 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:52.263681 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:52.263682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:52.263682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:52.263686 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:52.263697 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:52.263700 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:52.263749 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:52.263799 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 00:15:52.263812 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:15:52.266391 2971 catalog_manager.cc:1347] Generated new cluster ID: 672bca00c47540b69866492f427d743f
I20260812 00:15:52.266441 2971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:52.281620 2971 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:52.281783 2971 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:52.293923 2971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4543aea35bf54e14a4b385a1e7f9c90b: Generated new TSK 0
I20260812 00:15:52.294088 2971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:52.297523 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 2846
2026-08-12T00:15:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:52 dist-test-slave-zr1t krb5kdc[3000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:52 dist-test-slave-zr1t krb5kdc[3000](info): set up 2 sockets
Aug 12 00:15:52 dist-test-slave-zr1t krb5kdc[3000](info): commencing operation
krb5kdc: starting...
W20260812 00:15:53.569195 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:53 dist-test-slave-zr1t krb5kdc[3000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:53.684188 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:53.767449 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:53.767644 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:53.767666 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:53.768718 3025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:53.768750 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:53.768764 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:53.768777 3025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:53.768790 3025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:53.770282 3025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:53.770480 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:53.770640 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:53.775405 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:53.775369 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:53.775358 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:53.775538 3025 server_base.cc:1034] running on GCE node
I20260812 00:15:53.775836 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:53.776168 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:53.777331 3025 hybrid_clock.cc:648] HybridClock initialized: now 1786493753777309 us; error 34 us; skew 500 ppm
Aug 12 00:15:53 dist-test-slave-zr1t krb5kdc[3000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:53.779950 3025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:53.780421 3025 webserver.cc:460] Webserver started at http://127.0.0.1:36893/ using document root <none> and password file <none>
I20260812 00:15:53.780646 3025 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:53.780714 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:53.780874 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:53.781996 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "cd92f4303feb4f1fb457971cf9fcd87a"
format_stamp: "Formatted at 2026-08-12 00:15:53 on dist-test-slave-zr1t"
server_key: "975f933199763481d17a5ffcb0d63a07"
server_key_iv: "c4be0655e62261d5ce59118d1d96bffe"
server_key_version: "encryptionkey@0"
I20260812 00:15:53.782384 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "cd92f4303feb4f1fb457971cf9fcd87a"
format_stamp: "Formatted at 2026-08-12 00:15:53 on dist-test-slave-zr1t"
server_key: "975f933199763481d17a5ffcb0d63a07"
server_key_iv: "c4be0655e62261d5ce59118d1d96bffe"
server_key_version: "encryptionkey@0"
I20260812 00:15:53.783722 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:15:53.784457 3047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:53.784657 3025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:53.784766 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "cd92f4303feb4f1fb457971cf9fcd87a"
format_stamp: "Formatted at 2026-08-12 00:15:53 on dist-test-slave-zr1t"
server_key: "975f933199763481d17a5ffcb0d63a07"
server_key_iv: "c4be0655e62261d5ce59118d1d96bffe"
server_key_version: "encryptionkey@0"
I20260812 00:15:53.784842 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:53.802549 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:53.804970 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:53.805169 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:53.809473 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37497
I20260812 00:15:53.809573 3129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37497 every 8 connection(s)
I20260812 00:15:53.809929 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:53.810817 3130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:53.813100 3130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a: Bootstrap starting.
I20260812 00:15:53.813750 3130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:53.814059 3130 log.cc:824] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:53.814811 3130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a: No bootstrap required, opened a new log
I20260812 00:15:53.816368 3130 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37497 } }
I20260812 00:15:53.816509 3130 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:53.816532 3130 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd92f4303feb4f1fb457971cf9fcd87a, State: Initialized, Role: FOLLOWER
I20260812 00:15:53.816709 3130 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37497 } }
I20260812 00:15:53.816792 3130 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:53.816815 3130 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:53.816879 3130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:53.817396 3130 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37497 } }
I20260812 00:15:53.817512 3130 leader_election.cc:302] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd92f4303feb4f1fb457971cf9fcd87a; no voters:
I20260812 00:15:53.817711 3130 leader_election.cc:288] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:53.817777 3135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:53.817950 3135 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [term 1 LEADER]: Becoming Leader. State: Replica: cd92f4303feb4f1fb457971cf9fcd87a, State: Running, Role: LEADER
I20260812 00:15:53.817984 3130 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:53.818039 3135 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37497 } }
I20260812 00:15:53.818439 3137 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd92f4303feb4f1fb457971cf9fcd87a. Latest consensus state: current_term: 1 leader_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37497 } } }
I20260812 00:15:53.818529 3137 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:53.818581 3136 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd92f4303feb4f1fb457971cf9fcd87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37497 } } }
I20260812 00:15:53.818686 3136 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:53.819227 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3025
I20260812 00:15:53.819365 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:53.819572 32652 external_mini_cluster.cc:1428] Setting key bd75b91bb35c1eabfb5075d69afc102d
W20260812 00:15:53.819890 3160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:53.819964 3160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:53.820034 3148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:53.820222 3148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:53.821695 3148 catalog_manager.cc:1347] Generated new cluster ID: aeab9232d2f4415d96a75e5ab15438b7
I20260812 00:15:53.821753 3148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:15:53 dist-test-slave-zr1t krb5kdc[3000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:15:53.830588 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:53.820276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55189 (local address 127.0.0.1:37497)
0812 00:15:53.820449 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:53.820458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:53.820493 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:15:53.820720 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:53.820730 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:53.820791 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:53.820929 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:53.821735 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:53.822660 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:53.823728 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:53.824141 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:53.824228 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:53.826932 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:53.826936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:53.826936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:53.826949 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:53.828914 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:53.829339 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:53.829342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:53.829342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:53.829457 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:53.829704 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:53.829704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:53.829705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:53.829985 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:53.830128 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:53.830251 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 00:15:53.830376 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260812 00:15:53.831892 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:53.840281 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:53.833407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35509 (local address 127.0.0.1:37497)
0812 00:15:53.833447 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:53.833449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:53.833615 (+ 166us) server_negotiation.cc:407] Connection header received
0812 00:15:53.833656 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:53.833657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:53.833668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:53.833714 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:53.834301 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:53.835006 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:53.835970 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:53.836281 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:53.836335 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:53.837676 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:53.837677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:53.837678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:53.837685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:53.839194 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:53.839555 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:53.839556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:53.839556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:53.839602 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:53.839769 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:53.839769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:53.839770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:53.839871 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:53.839942 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:53.840126 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 00:15:53.840195 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:15:53.842563 3148 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:53.843029 3148 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:53.847625 3148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd92f4303feb4f1fb457971cf9fcd87a: Generated new TSK 0
I20260812 00:15:53.847793 3148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:53.851545 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3025
2026-08-12T00:15:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1545 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:53 dist-test-slave-zr1t krb5kdc[3179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:53 dist-test-slave-zr1t krb5kdc[3179](info): set up 2 sockets
Aug 12 00:15:53 dist-test-slave-zr1t krb5kdc[3179](info): commencing operation
krb5kdc: starting...
W20260812 00:15:55.119063 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:55 dist-test-slave-zr1t krb5kdc[3179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:55.235723 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:55.316406 3206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:55.316597 3206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:55.316619 3206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:55.317775 3206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:55.317814 3206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:55.317835 3206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:55.317874 3206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:55.317901 3206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:55.319288 3206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:55.319522 3206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:55.319732 3206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:55.324560 3214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:55.324556 3215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:55.324605 3206 server_base.cc:1034] running on GCE node
W20260812 00:15:55.324556 3218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:55.324896 3206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:55.325204 3206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:55.326360 3206 hybrid_clock.cc:648] HybridClock initialized: now 1786493755326341 us; error 30 us; skew 500 ppm
Aug 12 00:15:55 dist-test-slave-zr1t krb5kdc[3179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:55.336993 3206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:55.337343 3206 webserver.cc:460] Webserver started at http://127.0.0.1:38045/ using document root <none> and password file <none>
I20260812 00:15:55.337594 3206 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:55.337687 3206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:55.337791 3206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:55.338748 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "c336edd1a72b454794b7101f0a472512"
format_stamp: "Formatted at 2026-08-12 00:15:55 on dist-test-slave-zr1t"
server_key: "4d3149c65ec0a00effc80a9da9fc2d6a"
server_key_iv: "52dfcb05f98e6161c5858317fc735c56"
server_key_version: "encryptionkey@0"
I20260812 00:15:55.339046 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "c336edd1a72b454794b7101f0a472512"
format_stamp: "Formatted at 2026-08-12 00:15:55 on dist-test-slave-zr1t"
server_key: "4d3149c65ec0a00effc80a9da9fc2d6a"
server_key_iv: "52dfcb05f98e6161c5858317fc735c56"
server_key_version: "encryptionkey@0"
I20260812 00:15:55.340296 3206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:15:55.340965 3234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:55.341145 3206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:55.341231 3206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "c336edd1a72b454794b7101f0a472512"
format_stamp: "Formatted at 2026-08-12 00:15:55 on dist-test-slave-zr1t"
server_key: "4d3149c65ec0a00effc80a9da9fc2d6a"
server_key_iv: "52dfcb05f98e6161c5858317fc735c56"
server_key_version: "encryptionkey@0"
I20260812 00:15:55.341284 3206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:55.364915 3206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:55.366941 3206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:55.367055 3206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:55.371088 3206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46315
I20260812 00:15:55.371148 3325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46315 every 8 connection(s)
I20260812 00:15:55.371479 3206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:55.372411 3326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:55.374722 3326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512: Bootstrap starting.
I20260812 00:15:55.375272 3326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:55.375510 3326 log.cc:824] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:55.376147 3326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512: No bootstrap required, opened a new log
I20260812 00:15:55.377827 3326 raft_consensus.cc:348] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c336edd1a72b454794b7101f0a472512" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260812 00:15:55.377962 3326 raft_consensus.cc:374] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:55.377985 3326 raft_consensus.cc:729] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c336edd1a72b454794b7101f0a472512, State: Initialized, Role: FOLLOWER
I20260812 00:15:55.378113 3326 consensus_queue.cc:260] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c336edd1a72b454794b7101f0a472512" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260812 00:15:55.378196 3326 raft_consensus.cc:388] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:55.378219 3326 raft_consensus.cc:482] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:55.378247 3326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:55.378719 3326 raft_consensus.cc:504] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c336edd1a72b454794b7101f0a472512" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260812 00:15:55.378805 3326 leader_election.cc:302] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c336edd1a72b454794b7101f0a472512; no voters:
I20260812 00:15:55.378929 3326 leader_election.cc:288] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:55.379026 3331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:55.379155 3326 sys_catalog.cc:564] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:55.379228 3331 raft_consensus.cc:686] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [term 1 LEADER]: Becoming Leader. State: Replica: c336edd1a72b454794b7101f0a472512, State: Running, Role: LEADER
I20260812 00:15:55.379365 3331 consensus_queue.cc:237] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c336edd1a72b454794b7101f0a472512" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260812 00:15:55.379781 3333 sys_catalog.cc:455] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c336edd1a72b454794b7101f0a472512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c336edd1a72b454794b7101f0a472512" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260812 00:15:55.379866 3333 sys_catalog.cc:458] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:55.379907 3336 sys_catalog.cc:455] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c336edd1a72b454794b7101f0a472512. Latest consensus state: current_term: 1 leader_uuid: "c336edd1a72b454794b7101f0a472512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c336edd1a72b454794b7101f0a472512" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260812 00:15:55.379998 3336 sys_catalog.cc:458] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:55.380096 3349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:55.380656 3349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:55.380918 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3206
I20260812 00:15:55.381050 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:55.381242 32652 external_mini_cluster.cc:1428] Setting key 671b63ec74ea8a24d5e220b783d60740
I20260812 00:15:55.383862 3349 catalog_manager.cc:1347] Generated new cluster ID: d93227eed918478285eccdab0abc8e73
I20260812 00:15:55.383921 3349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:55.387053 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:55.381899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48567 (local address 127.0.0.1:46315)
0812 00:15:55.382080 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:55.382089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:55.382123 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:15:55.382236 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:55.382242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:55.382311 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:55.382455 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:55.382708 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:55.383697 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:55.384819 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:55.385418 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:55.385829 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:55.386241 (+ 412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:55.386245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:55.386245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:55.386441 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:55.386477 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:55.386489 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:55.386756 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:55.386798 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 00:15:55.386859 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260812 00:15:55.387868 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:55.392675 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:55.389289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52213 (local address 127.0.0.1:46315)
0812 00:15:55.389326 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:55.389327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:55.389438 (+ 111us) server_negotiation.cc:407] Connection header received
0812 00:15:55.389543 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:55.389544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:55.389554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:55.389580 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:55.389739 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:55.390439 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:55.391401 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:55.391695 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:55.391730 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:55.392450 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:55.392451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:55.392451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:55.392456 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:55.392469 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:55.392473 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:55.392498 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:55.392583 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 00:15:55.392604 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:15:55.398831 3349 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:55.399555 3349 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:55.405381 3349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c336edd1a72b454794b7101f0a472512: Generated new TSK 0
I20260812 00:15:55.405596 3349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:55.414089 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3206
2026-08-12T00:15:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1561 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:55 dist-test-slave-zr1t krb5kdc[3374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:55 dist-test-slave-zr1t krb5kdc[3374](info): set up 2 sockets
Aug 12 00:15:55 dist-test-slave-zr1t krb5kdc[3374](info): commencing operation
krb5kdc: starting...
W20260812 00:15:56.672216 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:56 dist-test-slave-zr1t krb5kdc[3374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:56.789639 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:56.870441 3395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:56.870661 3395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:56.870699 3395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:56.871857 3395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:56.871953 3395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:56.871980 3395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:56.872011 3395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:56.872035 3395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:56.873504 3395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:56.873835 3395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:56.874071 3395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:56.878845 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:56.878861 3410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:56.878839 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:56.878860 3395 server_base.cc:1034] running on GCE node
I20260812 00:15:56.879274 3395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:56.879559 3395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:56.880702 3395 hybrid_clock.cc:648] HybridClock initialized: now 1786493756880687 us; error 29 us; skew 500 ppm
Aug 12 00:15:56 dist-test-slave-zr1t krb5kdc[3374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:56.883122 3395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:56.883548 3395 webserver.cc:460] Webserver started at http://127.0.0.1:34741/ using document root <none> and password file <none>
I20260812 00:15:56.883783 3395 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:56.883849 3395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:56.884007 3395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:56.885038 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "4a00cf2b77d0413c88cf502f78cce3af"
format_stamp: "Formatted at 2026-08-12 00:15:56 on dist-test-slave-zr1t"
server_key: "fae225fa8e9ab728688df81a21c1f103"
server_key_iv: "932cbee6a83f33ae733d771b2f8aa97a"
server_key_version: "encryptionkey@0"
I20260812 00:15:56.885408 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "4a00cf2b77d0413c88cf502f78cce3af"
format_stamp: "Formatted at 2026-08-12 00:15:56 on dist-test-slave-zr1t"
server_key: "fae225fa8e9ab728688df81a21c1f103"
server_key_iv: "932cbee6a83f33ae733d771b2f8aa97a"
server_key_version: "encryptionkey@0"
I20260812 00:15:56.886740 3395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:15:56.887471 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:56.887651 3395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:15:56.887754 3395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "4a00cf2b77d0413c88cf502f78cce3af"
format_stamp: "Formatted at 2026-08-12 00:15:56 on dist-test-slave-zr1t"
server_key: "fae225fa8e9ab728688df81a21c1f103"
server_key_iv: "932cbee6a83f33ae733d771b2f8aa97a"
server_key_version: "encryptionkey@0"
I20260812 00:15:56.887840 3395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:56.891141 3395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:56.892951 3395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:56.893107 3395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:56.897261 3395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42159
I20260812 00:15:56.897291 3521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42159 every 8 connection(s)
I20260812 00:15:56.897673 3395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:56.898578 3522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:56.900839 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af: Bootstrap starting.
I20260812 00:15:56.901348 3522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:56.901676 3522 log.cc:824] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:56.902366 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af: No bootstrap required, opened a new log
I20260812 00:15:56.903800 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3395
I20260812 00:15:56.903930 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:56.904124 32652 external_mini_cluster.cc:1428] Setting key d0c80fd0a4b09d0242a7d2300bebdb29
I20260812 00:15:56.904089 3522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a00cf2b77d0413c88cf502f78cce3af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42159 } }
I20260812 00:15:56.904253 3522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:56.904702 3522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a00cf2b77d0413c88cf502f78cce3af, State: Initialized, Role: FOLLOWER
I20260812 00:15:56.904870 3522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a00cf2b77d0413c88cf502f78cce3af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42159 } }
I20260812 00:15:56.904968 3522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:56.904991 3522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:56.905062 3522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:56.906258 3522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a00cf2b77d0413c88cf502f78cce3af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42159 } }
I20260812 00:15:56.906388 3522 leader_election.cc:302] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a00cf2b77d0413c88cf502f78cce3af; no voters:
I20260812 00:15:56.906456 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:56.904652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.0.0.1:42159)
0812 00:15:56.904915 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:56.904923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:56.904948 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:15:56.905092 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:56.905100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:56.905131 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:56.905252 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:56.905971 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:56.905974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:56.905974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:56.905978 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:56.906014 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:56.906024 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:56.906080 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:56.906175 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 00:15:56.906227 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
I20260812 00:15:56.906553 3522 leader_election.cc:288] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:56.906628 3529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:56.906749 3529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [term 1 LEADER]: Becoming Leader. State: Replica: 4a00cf2b77d0413c88cf502f78cce3af, State: Running, Role: LEADER
I20260812 00:15:56.906809 3522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:56.907428 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:56.907529 3529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a00cf2b77d0413c88cf502f78cce3af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42159 } }
I20260812 00:15:56.908118 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a00cf2b77d0413c88cf502f78cce3af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a00cf2b77d0413c88cf502f78cce3af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42159 } } }
I20260812 00:15:56.908226 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:56.908633 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a00cf2b77d0413c88cf502f78cce3af. Latest consensus state: current_term: 1 leader_uuid: "4a00cf2b77d0413c88cf502f78cce3af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a00cf2b77d0413c88cf502f78cce3af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42159 } } }
I20260812 00:15:56.908741 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af [sys.catalog]: This master's current role is: LEADER
W20260812 00:15:56.908849 3556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:56.908916 3556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:56.909212 3557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:56.909510 3557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:15:56.910240 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:56.909089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39801 (local address 127.0.0.1:42159)
0812 00:15:56.909123 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:56.909125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:56.909271 (+ 146us) server_negotiation.cc:407] Connection header received
0812 00:15:56.909301 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:56.909302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:56.909318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:56.909393 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:56.910027 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:56.910028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:56.910028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:15:56.910032 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:56.910046 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:15:56.910051 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:56.910114 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:56.910174 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 00:15:56.910188 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 00:15:56.911064 3557 catalog_manager.cc:1347] Generated new cluster ID: c79143959fdc4bc08b3e93e6a83bd12e
I20260812 00:15:56.911114 3557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:56.931533 3557 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:56.931713 3557 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:56.940016 3557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a00cf2b77d0413c88cf502f78cce3af: Generated new TSK 0
I20260812 00:15:56.940212 3557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:56.947887 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3395
2026-08-12T00:15:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:57 dist-test-slave-zr1t krb5kdc[3580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:57 dist-test-slave-zr1t krb5kdc[3580](info): set up 2 sockets
Aug 12 00:15:57 dist-test-slave-zr1t krb5kdc[3580](info): commencing operation
krb5kdc: starting...
W20260812 00:15:58.224789 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:58 dist-test-slave-zr1t krb5kdc[3580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:58.341871 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:58.424175 3606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:58.424379 3606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:58.424402 3606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:58.425597 3606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:58.425640 3606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:58.425655 3606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:58.425668 3606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:58.425724 3606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:58.427239 3606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:58.427487 3606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:58.427717 3606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:58.432564 3617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:58.432580 3606 server_base.cc:1034] running on GCE node
W20260812 00:15:58.432538 3614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:58.432549 3615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:58.432996 3606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:58.433311 3606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:58.434484 3606 hybrid_clock.cc:648] HybridClock initialized: now 1786493758434451 us; error 58 us; skew 500 ppm
Aug 12 00:15:58 dist-test-slave-zr1t krb5kdc[3580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:58.437047 3606 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:58.437492 3606 webserver.cc:460] Webserver started at http://127.0.0.1:32783/ using document root <none> and password file <none>
I20260812 00:15:58.437734 3606 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:58.437778 3606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:58.437870 3606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:58.438850 3606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10"
format_stamp: "Formatted at 2026-08-12 00:15:58 on dist-test-slave-zr1t"
server_key: "8586cb050d5d32fdc9f6270560c9cbfe"
server_key_iv: "5a89065ba9ceeaa612a57a4dc0340aea"
server_key_version: "encryptionkey@0"
I20260812 00:15:58.439152 3606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10"
format_stamp: "Formatted at 2026-08-12 00:15:58 on dist-test-slave-zr1t"
server_key: "8586cb050d5d32fdc9f6270560c9cbfe"
server_key_iv: "5a89065ba9ceeaa612a57a4dc0340aea"
server_key_version: "encryptionkey@0"
I20260812 00:15:58.440405 3606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:15:58.441069 3632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:58.441239 3606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:15:58.441367 3606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10"
format_stamp: "Formatted at 2026-08-12 00:15:58 on dist-test-slave-zr1t"
server_key: "8586cb050d5d32fdc9f6270560c9cbfe"
server_key_iv: "5a89065ba9ceeaa612a57a4dc0340aea"
server_key_version: "encryptionkey@0"
I20260812 00:15:58.441447 3606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:58.455621 3606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:58.457412 3606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:58.457631 3606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:58.461501 3606 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260812 00:15:58.461591 3740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260812 00:15:58.461990 3606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:58.462847 3742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:15:58.465139 3742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10: Bootstrap starting.
I20260812 00:15:58.465781 3742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10: Neither blocks nor log segments found. Creating new log.
I20260812 00:15:58.466084 3742 log.cc:824] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10: Log is configured to *not* fsync() on all Append() calls
I20260812 00:15:58.466537 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3606
I20260812 00:15:58.466661 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:15:58.466874 32652 external_mini_cluster.cc:1428] Setting key aface12f277718d7e3dc0d2f4ae3e1d4
I20260812 00:15:58.466897 3742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10: No bootstrap required, opened a new log
I20260812 00:15:58.468647 3742 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260812 00:15:58.468806 3742 raft_consensus.cc:374] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:15:58.468870 3742 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82152ed4dcc4dc0a97dd7ed352fbe10, State: Initialized, Role: FOLLOWER
I20260812 00:15:58.469049 3742 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260812 00:15:58.469146 3742 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:15:58.469190 3742 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:15:58.469239 3742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:15:58.470201 3742 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260812 00:15:58.470322 3742 leader_election.cc:302] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82152ed4dcc4dc0a97dd7ed352fbe10; no voters:
I20260812 00:15:58.470515 3742 leader_election.cc:288] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:15:58.470537 3755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:15:58.470649 3755 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [term 1 LEADER]: Becoming Leader. State: Replica: e82152ed4dcc4dc0a97dd7ed352fbe10, State: Running, Role: LEADER
I20260812 00:15:58.470782 3742 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:15:58.470785 3755 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260812 00:15:58.471386 3757 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82152ed4dcc4dc0a97dd7ed352fbe10. Latest consensus state: current_term: 1 leader_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260812 00:15:58.471478 3757 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [sys.catalog]: This master's current role is: LEADER
I20260812 00:15:58.472980 3760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82152ed4dcc4dc0a97dd7ed352fbe10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260812 00:15:58.473136 3760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10 [sys.catalog]: This master's current role is: LEADER
W20260812 00:15:58.474550 3782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:15:58.474615 3782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:15:58.474680 3788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:15:58.474833 3788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:15:58 dist-test-slave-zr1t krb5kdc[3580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:15:58.476410 3788 catalog_manager.cc:1347] Generated new cluster ID: 3d281f34424d427389c48861684824ec
I20260812 00:15:58.476468 3788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:15:58.480516 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:58.467496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:53711 (local address 127.0.0.1:42683)
0812 00:15:58.467688 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:58.467697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:58.467732 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:15:58.468633 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:58.468641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:58.468692 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:58.468845 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:58.469651 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:58.470676 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:58.471691 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:58.473447 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:58.473591 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:58.476749 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:58.476753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:58.476754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:58.476765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:58.478767 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:58.479208 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:58.479212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:58.479212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:58.479357 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:58.479602 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:58.479603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:58.479603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:58.479936 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:58.480114 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:58.480197 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 00:15:58.480304 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260812 00:15:58.481709 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:15:58.488574 3788 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:15:58.489216 3788 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:15:58.493069 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:15:58.485834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:43115 (local address 127.0.0.1:42683)
0812 00:15:58.485872 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:15:58.485874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:15:58.485948 (+ 74us) server_negotiation.cc:407] Connection header received
0812 00:15:58.486007 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:15:58.486008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:15:58.486017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:15:58.486086 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:15:58.486691 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:58.487477 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:58.488423 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:15:58.488764 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:15:58.488845 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:15:58.490250 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:15:58.490251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:15:58.490252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:15:58.490259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:15:58.491855 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:58.492263 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:58.492264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:58.492265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:58.492327 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:15:58.492557 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:15:58.492558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:15:58.492558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:15:58.492687 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:15:58.492801 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 00:15:58.492910 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 00:15:58.492978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:15:58.499439 3788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e82152ed4dcc4dc0a97dd7ed352fbe10: Generated new TSK 0
I20260812 00:15:58.499634 3788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:15:58.503502 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3606
W20260812 00:15:58.518692 3576 connection.cc:535] client connection to 127.0.0.1:42683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42683): Connection reset by peer (error 104)
2026-08-12T00:15:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:15:58 dist-test-slave-zr1t krb5kdc[3807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:15:58 dist-test-slave-zr1t krb5kdc[3807](info): set up 2 sockets
Aug 12 00:15:58 dist-test-slave-zr1t krb5kdc[3807](info): commencing operation
krb5kdc: starting...
W20260812 00:15:59.768616 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:15:59 dist-test-slave-zr1t krb5kdc[3807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:15:59.882463 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:15:59.964903 3829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:15:59.965147 3829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:15:59.965185 3829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:15:59.966343 3829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:15:59.966420 3829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:15:59.966444 3829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:15:59.966475 3829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:15:59.966503 3829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:15:59.967931 3829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:15:59.968207 3829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:15:59.968447 3829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:15:59.973309 3844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:59.973400 3829 server_base.cc:1034] running on GCE node
W20260812 00:15:59.973387 3842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:15:59.973313 3841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:15:59.973884 3829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:15:59.974205 3829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:15:59.975375 3829 hybrid_clock.cc:648] HybridClock initialized: now 1786493759975369 us; error 38 us; skew 500 ppm
Aug 12 00:15:59 dist-test-slave-zr1t krb5kdc[3807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:15:59.977937 3829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:15:59.978407 3829 webserver.cc:460] Webserver started at http://127.0.0.1:42253/ using document root <none> and password file <none>
I20260812 00:15:59.978583 3829 fs_manager.cc:359] Metadata directory not provided
I20260812 00:15:59.978622 3829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:15:59.978708 3829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:15:59.979698 3829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "35f8096c0cbf44b09da7ace9e872185b"
format_stamp: "Formatted at 2026-08-12 00:15:59 on dist-test-slave-zr1t"
server_key: "462e1623bcc8b59646c66b5efa255814"
server_key_iv: "4f1cc952f9cb9c3860535451baff4070"
server_key_version: "encryptionkey@0"
I20260812 00:15:59.980031 3829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "35f8096c0cbf44b09da7ace9e872185b"
format_stamp: "Formatted at 2026-08-12 00:15:59 on dist-test-slave-zr1t"
server_key: "462e1623bcc8b59646c66b5efa255814"
server_key_iv: "4f1cc952f9cb9c3860535451baff4070"
server_key_version: "encryptionkey@0"
I20260812 00:15:59.981317 3829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:15:59.982110 3862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:15:59.982352 3829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:15:59.982445 3829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "35f8096c0cbf44b09da7ace9e872185b"
format_stamp: "Formatted at 2026-08-12 00:15:59 on dist-test-slave-zr1t"
server_key: "462e1623bcc8b59646c66b5efa255814"
server_key_iv: "4f1cc952f9cb9c3860535451baff4070"
server_key_version: "encryptionkey@0"
I20260812 00:15:59.982533 3829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:15:59.991093 3829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:15:59.992848 3829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:15:59.993009 3829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:15:59.996855 3829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34565
I20260812 00:15:59.996938 3957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34565 every 8 connection(s)
I20260812 00:15:59.997247 3829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:15:59.998106 3958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:00.000240 3958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b: Bootstrap starting.
I20260812 00:16:00.000765 3958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:00.001024 3958 log.cc:824] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:00.001749 3958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b: No bootstrap required, opened a new log
I20260812 00:16:00.003319 3958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f8096c0cbf44b09da7ace9e872185b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34565 } }
I20260812 00:16:00.003453 3958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:00.003475 3958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f8096c0cbf44b09da7ace9e872185b, State: Initialized, Role: FOLLOWER
I20260812 00:16:00.003608 3958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f8096c0cbf44b09da7ace9e872185b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34565 } }
I20260812 00:16:00.003691 3958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:00.003715 3958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:00.003743 3958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:00.004226 3958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f8096c0cbf44b09da7ace9e872185b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34565 } }
I20260812 00:16:00.004313 3958 leader_election.cc:302] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35f8096c0cbf44b09da7ace9e872185b; no voters:
I20260812 00:16:00.004441 3958 leader_election.cc:288] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:00.004503 3962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:00.004680 3962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [term 1 LEADER]: Becoming Leader. State: Replica: 35f8096c0cbf44b09da7ace9e872185b, State: Running, Role: LEADER
I20260812 00:16:00.004712 3958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:00.004789 3962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f8096c0cbf44b09da7ace9e872185b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34565 } }
I20260812 00:16:00.005142 3965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f8096c0cbf44b09da7ace9e872185b. Latest consensus state: current_term: 1 leader_uuid: "35f8096c0cbf44b09da7ace9e872185b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f8096c0cbf44b09da7ace9e872185b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34565 } } }
I20260812 00:16:00.005262 3963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35f8096c0cbf44b09da7ace9e872185b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f8096c0cbf44b09da7ace9e872185b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34565 } } }
I20260812 00:16:00.005285 3965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:00.005352 3963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:00.005770 3977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:00.006269 3977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:00.007284 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 3829
I20260812 00:16:00.007443 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:00.007658 32652 external_mini_cluster.cc:1428] Setting key 6c043c0996e29fbc6cec4174d00f723e
I20260812 00:16:00.007788 3977 catalog_manager.cc:1347] Generated new cluster ID: 3de0a6ba3d894f4a8506adf139514ac9
I20260812 00:16:00.007843 3977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:00.015381 3977 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:00.016059 3977 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:00.018362 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:00.008102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:36933 (local address 127.0.0.1:34565)
0812 00:16:00.013748 (+ 5646us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:00.013756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:00.013789 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:16:00.013933 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:00.013939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:00.013981 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:00.014106 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:00.014524 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:00.015612 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:00.016422 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:00.016864 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:00.016981 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:00.017697 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:00.017701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:00.017701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:00.017881 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:00.017910 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:00.017921 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:00.018016 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:00.018093 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 00:16:00.018138 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5602,"thread_start_us":74,"threads_started":1}
I20260812 00:16:00.019220 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:00.020939 3977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35f8096c0cbf44b09da7ace9e872185b: Generated new TSK 0
I20260812 00:16:00.021087 3977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:00.024098 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:00.020515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:40807 (local address 127.0.0.1:34565)
0812 00:16:00.020548 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:00.020549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:00.020799 (+ 250us) server_negotiation.cc:407] Connection header received
0812 00:16:00.020819 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:00.020820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:00.020831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:00.020910 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:00.021371 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:00.022138 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:00.022888 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:00.023192 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:00.023265 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:00.023860 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:00.023861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:00.023861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:00.023865 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:00.023884 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:00.023888 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:00.023941 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:00.024020 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:16:00.024032 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 00:16:00.025913 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 3829
W20260812 00:16:00.038558 3803 connection.cc:535] client connection to 127.0.0.1:34565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34565): Connection reset by peer (error 104)
2026-08-12T00:16:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:00 dist-test-slave-zr1t krb5kdc[4008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:00 dist-test-slave-zr1t krb5kdc[4008](info): set up 2 sockets
Aug 12 00:16:00 dist-test-slave-zr1t krb5kdc[4008](info): commencing operation
krb5kdc: starting...
W20260812 00:16:01.270150 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:01 dist-test-slave-zr1t krb5kdc[4008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:16:01.390453 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:01.473346 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:01.473627 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:01.473670 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:01.474772 4034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:01.474844 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:01.474869 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:01.474891 4034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:01.474918 4034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:01.476403 4034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:01.476701 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:01.476943 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:16:01.481982 4034 server_base.cc:1034] running on GCE node
W20260812 00:16:01.481963 4045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:01.481966 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:01.481994 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:01.482524 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:01.482780 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:01.483924 4034 hybrid_clock.cc:648] HybridClock initialized: now 1786493761483902 us; error 38 us; skew 500 ppm
Aug 12 00:16:01 dist-test-slave-zr1t krb5kdc[4008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:01.491364 4034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:16:01.491784 4034 webserver.cc:460] Webserver started at http://127.0.0.1:36149/ using document root <none> and password file <none>
I20260812 00:16:01.491992 4034 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:01.492031 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:01.492117 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:01.493165 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "903f60f5fe3e4541b710932dee2070f8"
format_stamp: "Formatted at 2026-08-12 00:16:01 on dist-test-slave-zr1t"
server_key: "56c63a899beb9d52ecff06b11fc4bbec"
server_key_iv: "69bf1d793dc8be36accf42cc9da03cbe"
server_key_version: "encryptionkey@0"
I20260812 00:16:01.493479 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "903f60f5fe3e4541b710932dee2070f8"
format_stamp: "Formatted at 2026-08-12 00:16:01 on dist-test-slave-zr1t"
server_key: "56c63a899beb9d52ecff06b11fc4bbec"
server_key_iv: "69bf1d793dc8be36accf42cc9da03cbe"
server_key_version: "encryptionkey@0"
I20260812 00:16:01.494853 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:16:01.495611 4063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:01.495781 4034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:16:01.495877 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "903f60f5fe3e4541b710932dee2070f8"
format_stamp: "Formatted at 2026-08-12 00:16:01 on dist-test-slave-zr1t"
server_key: "56c63a899beb9d52ecff06b11fc4bbec"
server_key_iv: "69bf1d793dc8be36accf42cc9da03cbe"
server_key_version: "encryptionkey@0"
I20260812 00:16:01.495935 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:01.499207 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:01.501402 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:01.501507 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:01.505466 4034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46727
I20260812 00:16:01.505559 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46727 every 8 connection(s)
I20260812 00:16:01.505880 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:01.506770 4156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:01.509074 4156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8: Bootstrap starting.
I20260812 00:16:01.509724 4156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:01.510043 4156 log.cc:824] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:01.510798 4156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8: No bootstrap required, opened a new log
I20260812 00:16:01.512542 4156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903f60f5fe3e4541b710932dee2070f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260812 00:16:01.512715 4156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:01.512768 4156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903f60f5fe3e4541b710932dee2070f8, State: Initialized, Role: FOLLOWER
I20260812 00:16:01.512943 4156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903f60f5fe3e4541b710932dee2070f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260812 00:16:01.513048 4156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:01.513092 4156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:01.513144 4156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:01.513747 4156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903f60f5fe3e4541b710932dee2070f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260812 00:16:01.513873 4156 leader_election.cc:302] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 903f60f5fe3e4541b710932dee2070f8; no voters:
I20260812 00:16:01.514045 4156 leader_election.cc:288] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:01.514094 4163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:01.514297 4163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [term 1 LEADER]: Becoming Leader. State: Replica: 903f60f5fe3e4541b710932dee2070f8, State: Running, Role: LEADER
I20260812 00:16:01.514321 4156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:01.514391 4163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903f60f5fe3e4541b710932dee2070f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260812 00:16:01.514744 4164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "903f60f5fe3e4541b710932dee2070f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903f60f5fe3e4541b710932dee2070f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } } }
I20260812 00:16:01.514751 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 903f60f5fe3e4541b710932dee2070f8. Latest consensus state: current_term: 1 leader_uuid: "903f60f5fe3e4541b710932dee2070f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903f60f5fe3e4541b710932dee2070f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } } }
I20260812 00:16:01.514842 4164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:01.514921 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:01.515141 4171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:01.515520 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4034
I20260812 00:16:01.515640 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:01.515748 4171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:01.515868 32652 external_mini_cluster.cc:1428] Setting key 7cec10a3b1c1b778c6d52c9b35ee91c6
I20260812 00:16:01.517566 4171 catalog_manager.cc:1347] Generated new cluster ID: 1e1474b7358a4dc6990851293f58f83c
I20260812 00:16:01.517623 4171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:01.519047 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:01.516341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:34645 (local address 127.0.0.1:46727)
0812 00:16:01.516566 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:01.516576 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:01.517288 (+ 712us) server_negotiation.cc:407] Connection header received
0812 00:16:01.517444 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:01.517452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:01.517482 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:01.517689 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:01.518611 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:01.518615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:01.518615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:01.518619 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:01.518666 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:01.518676 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:01.518745 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:01.518819 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 00:16:01.518867 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
I20260812 00:16:01.519873 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:01.522269 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:01.521171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:34285 (local address 127.0.0.1:46727)
0812 00:16:01.521205 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:01.521207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:01.521335 (+ 128us) server_negotiation.cc:407] Connection header received
0812 00:16:01.521403 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:01.521405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:01.521410 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:01.521451 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:01.522042 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:01.522043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:01.522043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:01.522048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:01.522062 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:01.522066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:01.522122 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:01.522177 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 00:16:01.522191 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:16:01.554838 4171 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:01.555074 4171 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:01.562943 4171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 903f60f5fe3e4541b710932dee2070f8: Generated new TSK 0
I20260812 00:16:01.563154 4171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:01.571995 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4034
2026-08-12T00:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1547 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:01 dist-test-slave-zr1t krb5kdc[4219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:01 dist-test-slave-zr1t krb5kdc[4219](info): set up 2 sockets
Aug 12 00:16:01 dist-test-slave-zr1t krb5kdc[4219](info): commencing operation
krb5kdc: starting...
W20260812 00:16:02.833509 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:02 dist-test-slave-zr1t krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:16:02.958727 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:03.042892 4247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:03.043151 4247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:03.043190 4247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:03.045274 4247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:03.045347 4247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:03.045378 4247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:03.045399 4247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:03.045423 4247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:03.048105 4247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:03.048378 4247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:03.048672 4247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:03.055066 4255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:03.055104 4254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:03.055056 4258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:03.055235 4247 server_base.cc:1034] running on GCE node
I20260812 00:16:03.055540 4247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:03.055852 4247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:03.057004 4247 hybrid_clock.cc:648] HybridClock initialized: now 1786493763056990 us; error 30 us; skew 500 ppm
Aug 12 00:16:03 dist-test-slave-zr1t krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:03.059583 4247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:16:03.060029 4247 webserver.cc:460] Webserver started at http://127.0.0.1:39793/ using document root <none> and password file <none>
I20260812 00:16:03.060257 4247 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:03.060309 4247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:03.060441 4247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:03.061510 4247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e"
format_stamp: "Formatted at 2026-08-12 00:16:03 on dist-test-slave-zr1t"
server_key: "b55aa498ed7f2b4f980c91174bf7b802"
server_key_iv: "57a3be34c72a87b52bf175095709f30f"
server_key_version: "encryptionkey@0"
I20260812 00:16:03.061937 4247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e"
format_stamp: "Formatted at 2026-08-12 00:16:03 on dist-test-slave-zr1t"
server_key: "b55aa498ed7f2b4f980c91174bf7b802"
server_key_iv: "57a3be34c72a87b52bf175095709f30f"
server_key_version: "encryptionkey@0"
I20260812 00:16:03.063313 4247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:03.064056 4270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:03.064249 4247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:16:03.064345 4247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e"
format_stamp: "Formatted at 2026-08-12 00:16:03 on dist-test-slave-zr1t"
server_key: "b55aa498ed7f2b4f980c91174bf7b802"
server_key_iv: "57a3be34c72a87b52bf175095709f30f"
server_key_version: "encryptionkey@0"
I20260812 00:16:03.064420 4247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:03.087899 4247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:03.089782 4247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:03.089915 4247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:03.094030 4247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39631
I20260812 00:16:03.094086 4363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39631 every 8 connection(s)
I20260812 00:16:03.094446 4247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:03.095295 4365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:03.097647 4365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e: Bootstrap starting.
I20260812 00:16:03.098241 4365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:03.098553 4365 log.cc:824] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:03.099260 4365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e: No bootstrap required, opened a new log
I20260812 00:16:03.100864 4365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39631 } }
I20260812 00:16:03.101003 4365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:03.101027 4365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fbcf5ffe5b4f6bb983ea093a568d5e, State: Initialized, Role: FOLLOWER
I20260812 00:16:03.101231 4365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39631 } }
I20260812 00:16:03.101327 4365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:03.101383 4365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:03.101438 4365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:03.102010 4365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39631 } }
I20260812 00:16:03.102134 4365 leader_election.cc:302] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28fbcf5ffe5b4f6bb983ea093a568d5e; no voters:
I20260812 00:16:03.102342 4365 leader_election.cc:288] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:03.102384 4371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:03.102583 4371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [term 1 LEADER]: Becoming Leader. State: Replica: 28fbcf5ffe5b4f6bb983ea093a568d5e, State: Running, Role: LEADER
I20260812 00:16:03.102615 4365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:03.102674 4371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39631 } }
I20260812 00:16:03.103093 4375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28fbcf5ffe5b4f6bb983ea093a568d5e. Latest consensus state: current_term: 1 leader_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39631 } } }
I20260812 00:16:03.103180 4375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:03.103224 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fbcf5ffe5b4f6bb983ea093a568d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39631 } } }
I20260812 00:16:03.103286 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:03.103430 4382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:03.103986 4382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:03.104326 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4247
I20260812 00:16:03.104465 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:03.104689 32652 external_mini_cluster.cc:1428] Setting key 9f708eb2c7550165b226bb3d61dd9228
I20260812 00:16:03.105816 4382 catalog_manager.cc:1347] Generated new cluster ID: 08c6a4ee830a48d9aa88da8353e8e91d
I20260812 00:16:03.105865 4382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:16:03 dist-test-slave-zr1t krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:16:03.115114 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:03.105391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:51363 (local address 127.0.0.1:39631)
0812 00:16:03.105566 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:03.105577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:03.105610 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:16:03.105799 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:03.105810 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:03.105888 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:03.106895 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:03.106967 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:03.107858 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:03.108618 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:03.109065 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:03.109187 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:03.111631 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:03.111635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:03.111635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:03.111647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:03.113369 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:03.113861 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:03.113865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:03.113865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:03.113966 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:03.114221 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:03.114222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:03.114222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:03.114492 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:03.114671 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:03.114763 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 00:16:03.114871 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260812 00:16:03.116125 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:03.123574 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:03.117453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:32951 (local address 127.0.0.1:39631)
0812 00:16:03.117488 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:03.117490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:03.117621 (+ 131us) server_negotiation.cc:407] Connection header received
0812 00:16:03.117701 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:03.117702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:03.117712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:03.117769 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:03.118268 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:03.118748 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:03.119447 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:03.119680 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:03.119756 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:03.120891 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:03.120893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:03.120893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:03.120900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:03.122400 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:03.122762 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:03.122764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:03.122764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:03.122818 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:03.123049 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:03.123050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:03.123051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:03.123161 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:03.123269 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:03.123396 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 00:16:03.123457 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:03.127269 4382 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:03.127853 4382 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:03.139452 4382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28fbcf5ffe5b4f6bb983ea093a568d5e: Generated new TSK 0
I20260812 00:16:03.139683 4382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:03.145375 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4247
W20260812 00:16:03.160310 4217 connection.cc:535] client connection to 127.0.0.1:39631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39631): Connection reset by peer (error 104)
2026-08-12T00:16:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1574 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:03 dist-test-slave-zr1t krb5kdc[4422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:03 dist-test-slave-zr1t krb5kdc[4422](info): set up 2 sockets
Aug 12 00:16:03 dist-test-slave-zr1t krb5kdc[4422](info): commencing operation
krb5kdc: starting...
W20260812 00:16:04.412149 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:04 dist-test-slave-zr1t krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:16:04.534720 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:04.618187 4447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:04.618436 4447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:04.618475 4447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:04.619643 4447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:04.619716 4447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:04.619740 4447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:04.619769 4447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:04.619807 4447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:04.621299 4447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:04.621592 4447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:04.621838 4447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:04.626878 4466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:04.626902 4468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:04.626885 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:04.627130 4447 server_base.cc:1034] running on GCE node
I20260812 00:16:04.627454 4447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:04.627777 4447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:04.628952 4447 hybrid_clock.cc:648] HybridClock initialized: now 1786493764628936 us; error 29 us; skew 500 ppm
Aug 12 00:16:04 dist-test-slave-zr1t krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:04.631572 4447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:16:04.632015 4447 webserver.cc:460] Webserver started at http://127.0.0.1:34785/ using document root <none> and password file <none>
I20260812 00:16:04.632269 4447 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:04.632347 4447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:04.632501 4447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:04.633682 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "42d90a9f423d4b50a4b22df2153c9d60"
format_stamp: "Formatted at 2026-08-12 00:16:04 on dist-test-slave-zr1t"
server_key: "6f20df668b991c38be417bef8d53bc06"
server_key_iv: "f28ec7051d2d12b8b11044fa8864b65d"
server_key_version: "encryptionkey@0"
I20260812 00:16:04.634057 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "42d90a9f423d4b50a4b22df2153c9d60"
format_stamp: "Formatted at 2026-08-12 00:16:04 on dist-test-slave-zr1t"
server_key: "6f20df668b991c38be417bef8d53bc06"
server_key_iv: "f28ec7051d2d12b8b11044fa8864b65d"
server_key_version: "encryptionkey@0"
I20260812 00:16:04.635546 4447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:04.636300 4477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:04.636492 4447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:16:04.636603 4447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "42d90a9f423d4b50a4b22df2153c9d60"
format_stamp: "Formatted at 2026-08-12 00:16:04 on dist-test-slave-zr1t"
server_key: "6f20df668b991c38be417bef8d53bc06"
server_key_iv: "f28ec7051d2d12b8b11044fa8864b65d"
server_key_version: "encryptionkey@0"
I20260812 00:16:04.636679 4447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:04.645327 4447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:04.647219 4447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:04.647393 4447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:04.651566 4447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39483
I20260812 00:16:04.651618 4568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39483 every 8 connection(s)
I20260812 00:16:04.651986 4447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:04.652864 4569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:04.655220 4569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60: Bootstrap starting.
I20260812 00:16:04.655792 4569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:04.656085 4569 log.cc:824] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:04.656780 4569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60: No bootstrap required, opened a new log
I20260812 00:16:04.658409 4569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d90a9f423d4b50a4b22df2153c9d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260812 00:16:04.658540 4569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:04.658561 4569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d90a9f423d4b50a4b22df2153c9d60, State: Initialized, Role: FOLLOWER
I20260812 00:16:04.658733 4569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d90a9f423d4b50a4b22df2153c9d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260812 00:16:04.658792 4569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:04.658838 4569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:04.658886 4569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:04.659420 4569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d90a9f423d4b50a4b22df2153c9d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260812 00:16:04.659531 4569 leader_election.cc:302] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d90a9f423d4b50a4b22df2153c9d60; no voters:
I20260812 00:16:04.659713 4569 leader_election.cc:288] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:04.659757 4578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:04.659942 4578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [term 1 LEADER]: Becoming Leader. State: Replica: 42d90a9f423d4b50a4b22df2153c9d60, State: Running, Role: LEADER
I20260812 00:16:04.660006 4569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:04.660122 4578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d90a9f423d4b50a4b22df2153c9d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260812 00:16:04.660416 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4447
I20260812 00:16:04.660557 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:04.660804 32652 external_mini_cluster.cc:1428] Setting key 450af54ca1b33612946b51c5a779962c
I20260812 00:16:04.661262 4584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42d90a9f423d4b50a4b22df2153c9d60. Latest consensus state: current_term: 1 leader_uuid: "42d90a9f423d4b50a4b22df2153c9d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d90a9f423d4b50a4b22df2153c9d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } } }
I20260812 00:16:04.661376 4584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:04.661597 4619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:04.661859 4581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42d90a9f423d4b50a4b22df2153c9d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d90a9f423d4b50a4b22df2153c9d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } } }
I20260812 00:16:04.661947 4581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:04.662377 4619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:04.663800 4619 catalog_manager.cc:1347] Generated new cluster ID: 5bd8cdce463c417988d4a97b6fbfc25f
I20260812 00:16:04.663862 4619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:04.667315 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:04.661640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:49217 (local address 127.0.0.1:39483)
0812 00:16:04.661823 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:04.661834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:04.661868 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:16:04.661990 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:04.661997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:04.662055 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:04.662210 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:04.662510 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:04.663920 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:04.665053 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:04.665735 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:04.665847 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:04.666618 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:04.666622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:04.666623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:04.666850 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:04.666888 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:04.666903 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:04.666988 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:04.667080 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 00:16:04.667108 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260812 00:16:04.668347 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:04.670717 4619 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:04.671192 4619 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:04.673988 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:04.670111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:54399 (local address 127.0.0.1:39483)
0812 00:16:04.670143 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:04.670145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:04.670158 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:16:04.670224 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:04.670225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:04.670252 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:04.670317 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:04.670622 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:04.671353 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:04.672465 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:04.672780 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:04.672849 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:04.673629 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:04.673631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:04.673631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:04.673635 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:04.673649 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:04.673654 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:04.673710 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:04.673822 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:16:04.673834 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:16:04.678966 4619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42d90a9f423d4b50a4b22df2153c9d60: Generated new TSK 0
I20260812 00:16:04.679149 4619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:04.683108 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4447
W20260812 00:16:04.712780 4417 connection.cc:535] client connection to 127.0.0.1:39483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39483): Connection reset by peer (error 104)
2026-08-12T00:16:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1552 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:04 dist-test-slave-zr1t krb5kdc[4640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:04 dist-test-slave-zr1t krb5kdc[4640](info): set up 2 sockets
Aug 12 00:16:04 dist-test-slave-zr1t krb5kdc[4640](info): commencing operation
krb5kdc: starting...
W20260812 00:16:05.955189 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:06 dist-test-slave-zr1t krb5kdc[4640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:16:06.076119 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:06.156942 4658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:06.157187 4658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:06.157224 4658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:06.158361 4658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:06.158434 4658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:06.158457 4658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:06.158478 4658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:06.158506 4658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:06.160003 4658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:06.160290 4658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:06.160528 4658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:06.165495 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:06.165505 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:06.165562 4658 server_base.cc:1034] running on GCE node
W20260812 00:16:06.165486 4668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:06.165992 4658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:06.166303 4658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:06.167488 4658 hybrid_clock.cc:648] HybridClock initialized: now 1786493766167474 us; error 44 us; skew 500 ppm
Aug 12 00:16:06 dist-test-slave-zr1t krb5kdc[4640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:06.170158 4658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:16:06.170619 4658 webserver.cc:460] Webserver started at http://127.0.0.1:35245/ using document root <none> and password file <none>
I20260812 00:16:06.170827 4658 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:06.170866 4658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:06.170954 4658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:06.171940 4658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "9723b499d8f04243af2f29ef0cd92ca2"
format_stamp: "Formatted at 2026-08-12 00:16:06 on dist-test-slave-zr1t"
server_key: "ffa44291be36d37c77aca37ebec364c4"
server_key_iv: "c0ae70da7755b108a86113430b1fca05"
server_key_version: "encryptionkey@0"
I20260812 00:16:06.172231 4658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "9723b499d8f04243af2f29ef0cd92ca2"
format_stamp: "Formatted at 2026-08-12 00:16:06 on dist-test-slave-zr1t"
server_key: "ffa44291be36d37c77aca37ebec364c4"
server_key_iv: "c0ae70da7755b108a86113430b1fca05"
server_key_version: "encryptionkey@0"
I20260812 00:16:06.173502 4658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:16:06.174266 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:06.174418 4658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:16:06.174517 4658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "9723b499d8f04243af2f29ef0cd92ca2"
format_stamp: "Formatted at 2026-08-12 00:16:06 on dist-test-slave-zr1t"
server_key: "ffa44291be36d37c77aca37ebec364c4"
server_key_iv: "c0ae70da7755b108a86113430b1fca05"
server_key_version: "encryptionkey@0"
I20260812 00:16:06.174572 4658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:06.177764 4658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:06.179499 4658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:06.179601 4658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:06.183512 4658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45621
I20260812 00:16:06.183588 4783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45621 every 8 connection(s)
I20260812 00:16:06.183897 4658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:06.184799 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:06.187078 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2: Bootstrap starting.
I20260812 00:16:06.187619 4786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:06.187860 4786 log.cc:824] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:06.188485 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2: No bootstrap required, opened a new log
I20260812 00:16:06.190107 4786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9723b499d8f04243af2f29ef0cd92ca2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45621 } }
I20260812 00:16:06.190241 4786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:06.190263 4786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9723b499d8f04243af2f29ef0cd92ca2, State: Initialized, Role: FOLLOWER
I20260812 00:16:06.190395 4786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9723b499d8f04243af2f29ef0cd92ca2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45621 } }
I20260812 00:16:06.190502 4786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:06.190544 4786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:06.190562 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4658
I20260812 00:16:06.190598 4786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:06.190706 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:06.190899 32652 external_mini_cluster.cc:1428] Setting key d58e68bb941cf9565d86895494e94eee
I20260812 00:16:06.191283 4786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9723b499d8f04243af2f29ef0cd92ca2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45621 } }
I20260812 00:16:06.191416 4786 leader_election.cc:302] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9723b499d8f04243af2f29ef0cd92ca2; no voters:
I20260812 00:16:06.191656 4786 leader_election.cc:288] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:06.191700 4794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:06.191884 4794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [term 1 LEADER]: Becoming Leader. State: Replica: 9723b499d8f04243af2f29ef0cd92ca2, State: Running, Role: LEADER
I20260812 00:16:06.191933 4786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:06.192057 4794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9723b499d8f04243af2f29ef0cd92ca2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45621 } }
I20260812 00:16:06.192468 4796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9723b499d8f04243af2f29ef0cd92ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9723b499d8f04243af2f29ef0cd92ca2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45621 } } }
I20260812 00:16:06.192553 4796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:06.192606 4798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9723b499d8f04243af2f29ef0cd92ca2. Latest consensus state: current_term: 1 leader_uuid: "9723b499d8f04243af2f29ef0cd92ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9723b499d8f04243af2f29ef0cd92ca2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45621 } } }
I20260812 00:16:06.192692 4798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:06.193292 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:06.191358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:33197 (local address 127.0.0.1:45621)
0812 00:16:06.191528 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:06.191537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:06.191578 (+ 41us) server_negotiation.cc:407] Connection header received
0812 00:16:06.191781 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:06.191788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:06.191819 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:06.192228 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:06.192726 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:06.192730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:06.192730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:06.192735 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:06.192810 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:06.192823 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:06.192912 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:06.193075 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 00:16:06.193123 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260812 00:16:06.194293 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 00:16:06.194319 4826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:06.194408 4826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:06.194469 4803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:06.194605 4803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:06.196095 4803 catalog_manager.cc:1347] Generated new cluster ID: 138b0328c94747bc8b3ae1fd8bccc84e
I20260812 00:16:06.196147 4803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:06.198733 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:06.195768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:42541 (local address 127.0.0.1:45621)
0812 00:16:06.195802 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:06.195804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:06.197758 (+ 1954us) server_negotiation.cc:407] Connection header received
0812 00:16:06.197842 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:06.197843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:06.197848 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:06.197903 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:06.198503 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:06.198504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:06.198504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:06.198508 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:06.198521 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:06.198525 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:06.198572 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:06.198626 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 00:16:06.198640 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:06.213846 4803 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:06.214040 4803 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:06.224218 4803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9723b499d8f04243af2f29ef0cd92ca2: Generated new TSK 0
I20260812 00:16:06.224421 4803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:06.230813 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4658
2026-08-12T00:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:06 dist-test-slave-zr1t krb5kdc[4848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:06 dist-test-slave-zr1t krb5kdc[4848](info): set up 2 sockets
Aug 12 00:16:06 dist-test-slave-zr1t krb5kdc[4848](info): commencing operation
krb5kdc: starting...
W20260812 00:16:07.498963 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:07 dist-test-slave-zr1t krb5kdc[4848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:07.620819 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40801
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:07.716434 4874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:07.716732 4874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:07.716773 4874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:07.718793 4874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:07.718863 4874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:07.718914 4874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:07.718946 4874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:07.718997 4874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:07.721498 4874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:40801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:07.722007 4874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:07.722314 4874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:07.728793 4892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:07.729074 4897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:07.728794 4888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:07.729115 4874 server_base.cc:1034] running on GCE node
I20260812 00:16:07.729472 4874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:07.729868 4874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:07.731076 4874 hybrid_clock.cc:648] HybridClock initialized: now 1786493767731052 us; error 41 us; skew 500 ppm
Aug 12 00:16:07 dist-test-slave-zr1t krb5kdc[4848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:07.733990 4874 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:07.734532 4874 webserver.cc:460] Webserver started at http://127.31.227.62:45251/ using document root <none> and password file <none>
I20260812 00:16:07.734823 4874 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:07.734900 4874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:07.735041 4874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:07.736254 4874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "db8ba64bdee1418394d205f74ae65764"
format_stamp: "Formatted at 2026-08-12 00:16:07 on dist-test-slave-zr1t"
server_key: "754c339297a7c0361a8b84c6c5d81c00"
server_key_iv: "51420f2aa9ed2fb903efb9524693a6ad"
server_key_version: "encryptionkey@0"
I20260812 00:16:07.736640 4874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "db8ba64bdee1418394d205f74ae65764"
format_stamp: "Formatted at 2026-08-12 00:16:07 on dist-test-slave-zr1t"
server_key: "754c339297a7c0361a8b84c6c5d81c00"
server_key_iv: "51420f2aa9ed2fb903efb9524693a6ad"
server_key_version: "encryptionkey@0"
I20260812 00:16:07.738646 4874 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 00:16:07.739626 4914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:07.739797 4874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:07.739919 4874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "db8ba64bdee1418394d205f74ae65764"
format_stamp: "Formatted at 2026-08-12 00:16:07 on dist-test-slave-zr1t"
server_key: "754c339297a7c0361a8b84c6c5d81c00"
server_key_iv: "51420f2aa9ed2fb903efb9524693a6ad"
server_key_version: "encryptionkey@0"
I20260812 00:16:07.739998 4874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:07.752959 4874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:07.755005 4874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:07.755177 4874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:07.759496 4874 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:40801
I20260812 00:16:07.759845 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:40801 every 8 connection(s)
I20260812 00:16:07.760075 4874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:07.760820 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:07.763314 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764: Bootstrap starting.
I20260812 00:16:07.763938 5023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:07.764245 5023 log.cc:824] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:07.764991 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764: No bootstrap required, opened a new log
I20260812 00:16:07.765970 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 4874
I20260812 00:16:07.766098 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:07.766319 32652 external_mini_cluster.cc:1428] Setting key 5f6619b8bd8dea1c30a1aeeceff2362a
I20260812 00:16:07.767145 5023 raft_consensus.cc:348] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8ba64bdee1418394d205f74ae65764" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40801 } }
I20260812 00:16:07.767313 5023 raft_consensus.cc:374] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:07.767367 5023 raft_consensus.cc:729] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8ba64bdee1418394d205f74ae65764, State: Initialized, Role: FOLLOWER
I20260812 00:16:07.767508 5023 consensus_queue.cc:260] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8ba64bdee1418394d205f74ae65764" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40801 } }
I20260812 00:16:07.767611 5023 raft_consensus.cc:388] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:07.767649 5023 raft_consensus.cc:482] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:07.767702 5023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:07.768271 5023 raft_consensus.cc:504] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8ba64bdee1418394d205f74ae65764" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40801 } }
I20260812 00:16:07.768419 5023 leader_election.cc:302] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db8ba64bdee1418394d205f74ae65764; no voters:
I20260812 00:16:07.768594 5023 leader_election.cc:288] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:07.768656 5028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:07.768778 5028 raft_consensus.cc:686] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [term 1 LEADER]: Becoming Leader. State: Replica: db8ba64bdee1418394d205f74ae65764, State: Running, Role: LEADER
I20260812 00:16:07.768893 5023 sys_catalog.cc:564] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:07.768869 5028 consensus_queue.cc:237] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8ba64bdee1418394d205f74ae65764" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40801 } }
I20260812 00:16:07.769225 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db8ba64bdee1418394d205f74ae65764. Latest consensus state: current_term: 1 leader_uuid: "db8ba64bdee1418394d205f74ae65764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8ba64bdee1418394d205f74ae65764" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40801 } } }
I20260812 00:16:07.769297 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:07.769474 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db8ba64bdee1418394d205f74ae65764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8ba64bdee1418394d205f74ae65764" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 40801 } } }
I20260812 00:16:07.769600 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:07.769778 5039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:07.770318 5039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:07.772152 5039 catalog_manager.cc:1347] Generated new cluster ID: 8d343acc54f741d691e0a4b134877e93
I20260812 00:16:07.772210 5039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:16:07 dist-test-slave-zr1t krb5kdc[4848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:07.777750 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:07.766861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43655 (local address 127.31.227.62:40801)
0812 00:16:07.767056 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:07.767065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:07.767091 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:16:07.767189 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:07.767195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:07.767238 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:07.767385 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:07.768188 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:07.769501 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:07.770778 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:07.771376 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:07.771483 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:07.774005 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:07.774009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:07.774010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:07.774022 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:07.775756 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:07.776176 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:07.776179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:07.776179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:07.776340 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:07.776578 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:07.776579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:07.776579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:07.776814 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:07.777385 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:07.777461 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 00:16:07.777606 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260812 00:16:07.778910 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:07.789703 5039 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:07.790409 5039 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:07.792038 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:07.780250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58323 (local address 127.31.227.62:40801)
0812 00:16:07.780289 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:07.780291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:07.785734 (+ 5443us) server_negotiation.cc:407] Connection header received
0812 00:16:07.785814 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:07.785815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:07.785825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:07.785890 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:07.786487 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:07.786977 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:07.787685 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:07.787901 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:07.787974 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:07.789250 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:07.789251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:07.789252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:07.789259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:07.790824 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:07.791193 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:07.791195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:07.791195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:07.791257 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:07.791501 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:07.791502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:07.791503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:07.791612 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:07.791734 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:07.791854 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 00:16:07.791919 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:16:07.798620 5039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db8ba64bdee1418394d205f74ae65764: Generated new TSK 0
I20260812 00:16:07.798789 5039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:07.803020 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 4874
W20260812 00:16:07.829564 4844 connection.cc:535] client connection to 127.31.227.62:40801 recv error: Network error: failed to read from TLS socket (remote: 127.31.227.62:40801): Connection reset by peer (error 104)
2026-08-12T00:16:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:07 dist-test-slave-zr1t krb5kdc[5082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:07 dist-test-slave-zr1t krb5kdc[5082](info): set up 2 sockets
Aug 12 00:16:07 dist-test-slave-zr1t krb5kdc[5082](info): commencing operation
krb5kdc: starting...
W20260812 00:16:09.099987 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:09 dist-test-slave-zr1t krb5kdc[5082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:09.218046 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:43669
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:09.302439 5103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:09.302628 5103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:09.302651 5103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:09.303783 5103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:09.303833 5103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:09.303848 5103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:09.303886 5103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:09.303915 5103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:09.305339 5103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:43669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:09.305639 5103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:09.305895 5103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:09.310846 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:09.310820 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:09.310851 5103 server_base.cc:1034] running on GCE node
W20260812 00:16:09.310820 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:09.311295 5103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:09.311595 5103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:09.312736 5103 hybrid_clock.cc:648] HybridClock initialized: now 1786493769312718 us; error 42 us; skew 500 ppm
Aug 12 00:16:09 dist-test-slave-zr1t krb5kdc[5082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493769, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:09.315264 5103 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:09.315655 5103 webserver.cc:460] Webserver started at http://127.31.227.62:38111/ using document root <none> and password file <none>
I20260812 00:16:09.315841 5103 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:09.315881 5103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:09.315971 5103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:09.316977 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "6604dd5d766f4dbfb4cc749580f75cc9"
format_stamp: "Formatted at 2026-08-12 00:16:09 on dist-test-slave-zr1t"
server_key: "47677d0f5ff781bcf90d794b5bfb4a9e"
server_key_iv: "3125dbaf2ff932db29845e38db4a69d0"
server_key_version: "encryptionkey@0"
I20260812 00:16:09.317271 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "6604dd5d766f4dbfb4cc749580f75cc9"
format_stamp: "Formatted at 2026-08-12 00:16:09 on dist-test-slave-zr1t"
server_key: "47677d0f5ff781bcf90d794b5bfb4a9e"
server_key_iv: "3125dbaf2ff932db29845e38db4a69d0"
server_key_version: "encryptionkey@0"
I20260812 00:16:09.318638 5103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:16:09.319365 5126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:09.319558 5103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:09.319648 5103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "6604dd5d766f4dbfb4cc749580f75cc9"
format_stamp: "Formatted at 2026-08-12 00:16:09 on dist-test-slave-zr1t"
server_key: "47677d0f5ff781bcf90d794b5bfb4a9e"
server_key_iv: "3125dbaf2ff932db29845e38db4a69d0"
server_key_version: "encryptionkey@0"
I20260812 00:16:09.319706 5103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:09.342144 5103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:09.344565 5103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:09.344679 5103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:09.348647 5103 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:43669
I20260812 00:16:09.348711 5224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:43669 every 8 connection(s)
I20260812 00:16:09.349092 5103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:09.350030 5225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:09.352330 5225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9: Bootstrap starting.
I20260812 00:16:09.352901 5225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:09.353252 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5103
I20260812 00:16:09.353317 5225 log.cc:824] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:09.353394 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:09.353641 32652 external_mini_cluster.cc:1428] Setting key 6d4d572575ddab96d327536171d160b4
I20260812 00:16:09.354163 5225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9: No bootstrap required, opened a new log
I20260812 00:16:09.355810 5225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43669 } }
I20260812 00:16:09.355954 5225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:09.355978 5225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6604dd5d766f4dbfb4cc749580f75cc9, State: Initialized, Role: FOLLOWER
I20260812 00:16:09.356138 5225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43669 } }
I20260812 00:16:09.356228 5225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:09.356287 5225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:09.356345 5225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:09.357106 5225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43669 } }
I20260812 00:16:09.357221 5225 leader_election.cc:302] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6604dd5d766f4dbfb4cc749580f75cc9; no voters:
I20260812 00:16:09.357374 5225 leader_election.cc:288] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:09.357432 5237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:09.357607 5237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 6604dd5d766f4dbfb4cc749580f75cc9, State: Running, Role: LEADER
I20260812 00:16:09.357657 5225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:09.357694 5237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43669 } }
I20260812 00:16:09.358042 5238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43669 } } }
I20260812 00:16:09.358028 5237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6604dd5d766f4dbfb4cc749580f75cc9. Latest consensus state: current_term: 1 leader_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604dd5d766f4dbfb4cc749580f75cc9" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 43669 } } }
I20260812 00:16:09.358136 5238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:09.358141 5237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:09.358786 5246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:09.359289 5246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:09.361958 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:09.354190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53811 (local address 127.31.227.62:43669)
0812 00:16:09.354401 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:09.354410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:09.354445 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:16:09.355826 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:09.355835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:09.355891 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:09.356058 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:09.356372 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:09.358501 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:09.359631 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:09.360238 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:09.360346 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:09.361098 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:09.361102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:09.361102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:09.361325 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:09.361372 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:09.361389 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:09.361484 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:09.361701 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 00:16:09.361731 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260812 00:16:09.363004 5246 catalog_manager.cc:1347] Generated new cluster ID: 629be5e8901f403a8a0f7247a6bf1aaf
I20260812 00:16:09.362996 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:09.363054 5246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:09.367641 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:09.364429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35119 (local address 127.31.227.62:43669)
0812 00:16:09.364486 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:09.364488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:09.364595 (+ 107us) server_negotiation.cc:407] Connection header received
0812 00:16:09.364644 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:09.364644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:09.364654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:09.364699 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:09.364801 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:09.365280 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:09.366253 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:09.366605 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:09.366684 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:09.367387 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:09.367388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:09.367389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:09.367393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:09.367410 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:09.367415 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:09.367483 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:09.367553 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 00:16:09.367566 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:16:09.376356 5246 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:09.377056 5246 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:09.392951 5246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6604dd5d766f4dbfb4cc749580f75cc9: Generated new TSK 0
I20260812 00:16:09.393162 5246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:09.400703 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5103
W20260812 00:16:09.418138 5080 connection.cc:535] client connection to 127.31.227.62:43669 recv error: Network error: failed to read from TLS socket (remote: 127.31.227.62:43669): Connection reset by peer (error 104)
2026-08-12T00:16:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:09 dist-test-slave-zr1t krb5kdc[5278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:09 dist-test-slave-zr1t krb5kdc[5278](info): set up 2 sockets
Aug 12 00:16:09 dist-test-slave-zr1t krb5kdc[5278](info): commencing operation
krb5kdc: starting...
W20260812 00:16:10.671968 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:10 dist-test-slave-zr1t krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:10.791500 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42783
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:10.873705 5301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:10.873934 5301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:10.873972 5301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:10.875170 5301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:10.875237 5301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:10.875262 5301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:10.875283 5301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:10.875337 5301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:10.876827 5301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:10.877117 5301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:10.877351 5301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:10.882333 5313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:10.882304 5316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:10.882303 5312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:10.882627 5301 server_base.cc:1034] running on GCE node
I20260812 00:16:10.882858 5301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:10.883090 5301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:10.884300 5301 hybrid_clock.cc:648] HybridClock initialized: now 1786493770884262 us; error 29 us; skew 500 ppm
Aug 12 00:16:10 dist-test-slave-zr1t krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493770, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:10.886976 5301 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:10.887360 5301 webserver.cc:460] Webserver started at http://127.31.227.62:41539/ using document root <none> and password file <none>
I20260812 00:16:10.887542 5301 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:10.887583 5301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:10.887682 5301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:10.888622 5301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "88e98a32d14e4e408af05d4833cfa1ba"
format_stamp: "Formatted at 2026-08-12 00:16:10 on dist-test-slave-zr1t"
server_key: "c7fee86135583251f50fdb5321207816"
server_key_iv: "60f2c59cc0000ce51b86a90154ee3003"
server_key_version: "encryptionkey@0"
I20260812 00:16:10.888906 5301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "88e98a32d14e4e408af05d4833cfa1ba"
format_stamp: "Formatted at 2026-08-12 00:16:10 on dist-test-slave-zr1t"
server_key: "c7fee86135583251f50fdb5321207816"
server_key_iv: "60f2c59cc0000ce51b86a90154ee3003"
server_key_version: "encryptionkey@0"
I20260812 00:16:10.890184 5301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:16:10.890833 5330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:10.891043 5301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:16:10.891117 5301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "88e98a32d14e4e408af05d4833cfa1ba"
format_stamp: "Formatted at 2026-08-12 00:16:10 on dist-test-slave-zr1t"
server_key: "c7fee86135583251f50fdb5321207816"
server_key_iv: "60f2c59cc0000ce51b86a90154ee3003"
server_key_version: "encryptionkey@0"
I20260812 00:16:10.891170 5301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:10.894330 5301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:10.896653 5301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:10.896759 5301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:10.900774 5301 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42783
I20260812 00:16:10.900856 5421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42783 every 8 connection(s)
I20260812 00:16:10.901139 5301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:10.902145 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:10.904484 5423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba: Bootstrap starting.
I20260812 00:16:10.905038 5423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:10.905333 5423 log.cc:824] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:10.905961 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5301
I20260812 00:16:10.906046 5423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba: No bootstrap required, opened a new log
I20260812 00:16:10.906083 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:10.906256 32652 external_mini_cluster.cc:1428] Setting key edd4c24b1f72187bdf25f1790b0a523c
I20260812 00:16:10.908423 5423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e98a32d14e4e408af05d4833cfa1ba" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42783 } }
I20260812 00:16:10.908596 5423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:10.908670 5423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e98a32d14e4e408af05d4833cfa1ba, State: Initialized, Role: FOLLOWER
I20260812 00:16:10.908840 5423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e98a32d14e4e408af05d4833cfa1ba" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42783 } }
I20260812 00:16:10.908946 5423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:10.908986 5423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:10.909075 5423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:10.910223 5423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e98a32d14e4e408af05d4833cfa1ba" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42783 } }
I20260812 00:16:10.910352 5423 leader_election.cc:302] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e98a32d14e4e408af05d4833cfa1ba; no voters:
I20260812 00:16:10.910527 5423 leader_election.cc:288] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:10.910563 5429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:10.910796 5423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:10.910770 5429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [term 1 LEADER]: Becoming Leader. State: Replica: 88e98a32d14e4e408af05d4833cfa1ba, State: Running, Role: LEADER
I20260812 00:16:10.910990 5429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e98a32d14e4e408af05d4833cfa1ba" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42783 } }
I20260812 00:16:10.911052 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:10.906943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38911 (local address 127.31.227.62:42783)
0812 00:16:10.907439 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:10.907449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:10.907482 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:16:10.908372 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:10.908382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:10.908420 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:10.908563 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:10.910336 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:10.910339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:10.910339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:10.910344 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:10.910392 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:10.910407 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:10.910485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:10.910724 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 00:16:10.910765 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":66,"threads_started":1}
I20260812 00:16:10.911376 5429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e98a32d14e4e408af05d4833cfa1ba. Latest consensus state: current_term: 1 leader_uuid: "88e98a32d14e4e408af05d4833cfa1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e98a32d14e4e408af05d4833cfa1ba" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42783 } } }
I20260812 00:16:10.911464 5429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:10.911617 5432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e98a32d14e4e408af05d4833cfa1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e98a32d14e4e408af05d4833cfa1ba" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42783 } } }
I20260812 00:16:10.911675 5444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:10.911713 5432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:10.912137 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:10.912223 5444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:10.914927 5444 catalog_manager.cc:1347] Generated new cluster ID: c3de7f08d84f4365a3072facd435da1f
I20260812 00:16:10.914929 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:10.913668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56345 (local address 127.31.227.62:42783)
0812 00:16:10.913704 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:10.913706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:10.913969 (+ 263us) server_negotiation.cc:407] Connection header received
0812 00:16:10.914041 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:10.914042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:10.914048 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:10.914102 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:10.914699 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:10.914701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:10.914701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:10.914705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:10.914719 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:10.914723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:10.914751 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:10.914860 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 00:16:10.914875 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:10.914990 5444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:10.931670 5444 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:10.931860 5444 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:10.943416 5444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e98a32d14e4e408af05d4833cfa1ba: Generated new TSK 0
I20260812 00:16:10.943619 5444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:10.947144 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5301
2026-08-12T00:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:11 dist-test-slave-zr1t krb5kdc[5469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:11 dist-test-slave-zr1t krb5kdc[5469](info): set up 2 sockets
Aug 12 00:16:11 dist-test-slave-zr1t krb5kdc[5469](info): commencing operation
krb5kdc: starting...
W20260812 00:16:12.217974 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:12 dist-test-slave-zr1t krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:12.337914 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42183
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:12.419286 5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:12.419503 5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:12.419528 5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:12.420675 5495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:12.420718 5495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:12.420732 5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:12.420744 5495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:12.420794 5495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:12.422325 5495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:12.422567 5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:12.422792 5495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:12.427851 5506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:12.427842 5509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:12.427925 5495 server_base.cc:1034] running on GCE node
W20260812 00:16:12.427851 5517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:12.428330 5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:12.428629 5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:12.429761 5495 hybrid_clock.cc:648] HybridClock initialized: now 1786493772429745 us; error 27 us; skew 500 ppm
Aug 12 00:16:12 dist-test-slave-zr1t krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:12.432456 5495 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:12.432945 5495 webserver.cc:460] Webserver started at http://127.31.227.62:35045/ using document root <none> and password file <none>
I20260812 00:16:12.433182 5495 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:12.433228 5495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:12.433408 5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:12.434516 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "241946fb96df46ee959381207623bfe3"
format_stamp: "Formatted at 2026-08-12 00:16:12 on dist-test-slave-zr1t"
server_key: "f505139c33daf8628eb19a9cb36bad94"
server_key_iv: "0539cd261b4077f00db3d0c089200fb9"
server_key_version: "encryptionkey@0"
I20260812 00:16:12.434886 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "241946fb96df46ee959381207623bfe3"
format_stamp: "Formatted at 2026-08-12 00:16:12 on dist-test-slave-zr1t"
server_key: "f505139c33daf8628eb19a9cb36bad94"
server_key_iv: "0539cd261b4077f00db3d0c089200fb9"
server_key_version: "encryptionkey@0"
I20260812 00:16:12.436358 5495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:16:12.437153 5536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:12.437353 5495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:12.437469 5495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "241946fb96df46ee959381207623bfe3"
format_stamp: "Formatted at 2026-08-12 00:16:12 on dist-test-slave-zr1t"
server_key: "f505139c33daf8628eb19a9cb36bad94"
server_key_iv: "0539cd261b4077f00db3d0c089200fb9"
server_key_version: "encryptionkey@0"
I20260812 00:16:12.437572 5495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:12.452097 5495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:12.454111 5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:12.454280 5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:12.458738 5495 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42183
I20260812 00:16:12.458793 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42183 every 8 connection(s)
I20260812 00:16:12.459193 5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:12.460093 5644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:12.462553 5644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3: Bootstrap starting.
I20260812 00:16:12.462590 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5495
I20260812 00:16:12.462695 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:12.462905 32652 external_mini_cluster.cc:1428] Setting key df2f39b619f0d248a49bb0b6994187be
I20260812 00:16:12.463304 5644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:12.463691 5644 log.cc:824] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:12.464474 5644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3: No bootstrap required, opened a new log
I20260812 00:16:12.466926 5644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241946fb96df46ee959381207623bfe3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42183 } }
I20260812 00:16:12.467098 5644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:12.467131 5644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241946fb96df46ee959381207623bfe3, State: Initialized, Role: FOLLOWER
I20260812 00:16:12.467248 5644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241946fb96df46ee959381207623bfe3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42183 } }
I20260812 00:16:12.467320 5644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:12.467351 5644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:12.467397 5644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:12.467999 5644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241946fb96df46ee959381207623bfe3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42183 } }
I20260812 00:16:12.468117 5644 leader_election.cc:302] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241946fb96df46ee959381207623bfe3; no voters:
I20260812 00:16:12.468297 5644 leader_election.cc:288] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:12.468346 5649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:12.468528 5649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [term 1 LEADER]: Becoming Leader. State: Replica: 241946fb96df46ee959381207623bfe3, State: Running, Role: LEADER
I20260812 00:16:12.468618 5644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:12.468676 5649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241946fb96df46ee959381207623bfe3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42183 } }
I20260812 00:16:12.469034 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241946fb96df46ee959381207623bfe3. Latest consensus state: current_term: 1 leader_uuid: "241946fb96df46ee959381207623bfe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241946fb96df46ee959381207623bfe3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42183 } } }
I20260812 00:16:12.469131 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [sys.catalog]: This master's current role is: LEADER
Aug 12 00:16:12 dist-test-slave-zr1t krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:12.470547 5651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241946fb96df46ee959381207623bfe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241946fb96df46ee959381207623bfe3" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42183 } } }
I20260812 00:16:12.470909 5651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3 [sys.catalog]: This master's current role is: LEADER
W20260812 00:16:12.470662 5680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:12.471025 5680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:12.471113 5656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:12.471305 5656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:12.472851 5656 catalog_manager.cc:1347] Generated new cluster ID: 165ad32a33a14b18adcb5f5c3ebe7b17
I20260812 00:16:12.472916 5656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:12.475055 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:12.463466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54587 (local address 127.31.227.62:42183)
0812 00:16:12.463677 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:12.463690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:12.463717 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:16:12.464343 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:12.464350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:12.464408 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:12.464539 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:12.465429 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:12.466384 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:12.467490 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:12.467902 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:12.467992 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:12.471320 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:12.471324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:12.471324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:12.471334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:12.473370 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:12.473784 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:12.473787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:12.473788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:12.473910 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:12.474125 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:12.474126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:12.474126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:12.474433 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:12.474596 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:12.474697 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 00:16:12.474831 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260812 00:16:12.476197 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:12.488871 5656 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:12.488966 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:12.477836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43539 (local address 127.31.227.62:42183)
0812 00:16:12.477879 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:12.477881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:12.481687 (+ 3806us) server_negotiation.cc:407] Connection header received
0812 00:16:12.481719 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:12.481720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:12.481730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:12.481796 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:12.482390 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:12.483165 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:12.484115 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:12.484450 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:12.484522 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:12.485823 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:12.485824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:12.485825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:12.485832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:12.487407 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:12.487801 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:12.487803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:12.487803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:12.487862 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:12.488081 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:12.488082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:12.488082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:12.488205 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:12.488303 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:12.488732 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 00:16:12.488807 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:12.489691 5656 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:12.491641 5656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241946fb96df46ee959381207623bfe3: Generated new TSK 0
I20260812 00:16:12.491822 5656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:12.496248 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5495
2026-08-12T00:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:12 dist-test-slave-zr1t krb5kdc[5696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:12 dist-test-slave-zr1t krb5kdc[5696](info): set up 2 sockets
Aug 12 00:16:12 dist-test-slave-zr1t krb5kdc[5696](info): commencing operation
krb5kdc: starting...
W20260812 00:16:13.769905 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:13 dist-test-slave-zr1t krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:13.889825 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42543
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:13.974722 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:13.974920 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:13.974944 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:13.976046 5719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:13.976082 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:13.976097 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:13.976109 5719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:13.976121 5719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:13.977533 5719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:13.977722 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:13.977921 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:13.982978 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:13.982995 5731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:13.983004 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:13.983388 5719 server_base.cc:1034] running on GCE node
I20260812 00:16:13.983656 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:13.983970 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:13.985113 5719 hybrid_clock.cc:648] HybridClock initialized: now 1786493773985093 us; error 37 us; skew 500 ppm
Aug 12 00:16:13 dist-test-slave-zr1t krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493773, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:13.992741 5719 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:13.993206 5719 webserver.cc:460] Webserver started at http://127.31.227.62:36669/ using document root <none> and password file <none>
I20260812 00:16:13.993485 5719 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:13.993606 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:13.993762 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:13.994868 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "224c2bcbbbd84f2ca8c50658d5248f86"
format_stamp: "Formatted at 2026-08-12 00:16:13 on dist-test-slave-zr1t"
server_key: "b40d9149292661ba44e01b3f38f39e39"
server_key_iv: "eedf6961072e187725ad049870d54a3e"
server_key_version: "encryptionkey@0"
I20260812 00:16:13.995227 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "224c2bcbbbd84f2ca8c50658d5248f86"
format_stamp: "Formatted at 2026-08-12 00:16:13 on dist-test-slave-zr1t"
server_key: "b40d9149292661ba44e01b3f38f39e39"
server_key_iv: "eedf6961072e187725ad049870d54a3e"
server_key_version: "encryptionkey@0"
I20260812 00:16:13.996603 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:16:13.997316 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:13.997579 5719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:13.997680 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "224c2bcbbbd84f2ca8c50658d5248f86"
format_stamp: "Formatted at 2026-08-12 00:16:13 on dist-test-slave-zr1t"
server_key: "b40d9149292661ba44e01b3f38f39e39"
server_key_iv: "eedf6961072e187725ad049870d54a3e"
server_key_version: "encryptionkey@0"
I20260812 00:16:13.997762 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:14.036204 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:14.038137 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:14.038300 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:14.042399 5719 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42543
I20260812 00:16:14.042438 5855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42543 every 8 connection(s)
I20260812 00:16:14.042879 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:14.043745 5857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:14.045307 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5719
I20260812 00:16:14.045423 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:14.045645 32652 external_mini_cluster.cc:1428] Setting key 9e27bb63030c4b906eca311512d9b413
I20260812 00:16:14.046414 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86: Bootstrap starting.
I20260812 00:16:14.047092 5857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:14.047435 5857 log.cc:824] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:14.048666 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86: No bootstrap required, opened a new log
I20260812 00:16:14.051105 5857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42543 } }
I20260812 00:16:14.051272 5857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:14.051339 5857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224c2bcbbbd84f2ca8c50658d5248f86, State: Initialized, Role: FOLLOWER
I20260812 00:16:14.051411 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:14.046195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54085 (local address 127.31.227.62:42543)
0812 00:16:14.046396 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:14.046405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:14.046438 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:16:14.047042 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:14.047051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:14.047114 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:14.047273 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:14.047494 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:14.048458 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:14.049299 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:14.049930 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:14.050038 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:14.050699 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:14.050704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:14.050704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:14.050937 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:14.050975 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:14.050990 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:14.051053 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:14.051117 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 00:16:14.051179 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260812 00:16:14.051470 5857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42543 } }
I20260812 00:16:14.051559 5857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:14.051599 5857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:14.051645 5857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:14.052362 5857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42543 } }
I20260812 00:16:14.052392 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:14.052484 5857 leader_election.cc:302] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224c2bcbbbd84f2ca8c50658d5248f86; no voters:
I20260812 00:16:14.052649 5857 leader_election.cc:288] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:14.052688 5869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:14.052855 5869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [term 1 LEADER]: Becoming Leader. State: Replica: 224c2bcbbbd84f2ca8c50658d5248f86, State: Running, Role: LEADER
I20260812 00:16:14.052923 5857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:14.053045 5869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42543 } }
I20260812 00:16:14.053429 5869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224c2bcbbbd84f2ca8c50658d5248f86. Latest consensus state: current_term: 1 leader_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42543 } } }
I20260812 00:16:14.053442 5879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224c2bcbbbd84f2ca8c50658d5248f86" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42543 } } }
I20260812 00:16:14.053670 5879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:14.053660 5869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:14.053915 5897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:14.054541 5897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:14.056252 5897 catalog_manager.cc:1347] Generated new cluster ID: 76dd3d84e03c4306a401ccab47bc1602
I20260812 00:16:14.056303 5897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:14.057302 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:14.053973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44435 (local address 127.31.227.62:42543)
0812 00:16:14.054012 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:14.054014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:14.054208 (+ 194us) server_negotiation.cc:407] Connection header received
0812 00:16:14.054287 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:14.054288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:14.054299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:14.054349 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:14.054530 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:14.055173 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:14.056113 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:14.056393 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:14.056437 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:14.057090 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:14.057091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:14.057092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:14.057095 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:14.057107 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:14.057111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:14.057134 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:14.057217 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 00:16:14.057237 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:16:14.071030 5897 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:14.071585 5897 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:14.080466 5897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224c2bcbbbd84f2ca8c50658d5248f86: Generated new TSK 0
I20260812 00:16:14.080623 5897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:14.084707 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5719
2026-08-12T00:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:14 dist-test-slave-zr1t krb5kdc[5928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:14 dist-test-slave-zr1t krb5kdc[5928](info): set up 2 sockets
Aug 12 00:16:14 dist-test-slave-zr1t krb5kdc[5928](info): commencing operation
krb5kdc: starting...
W20260812 00:16:15.361691 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:15 dist-test-slave-zr1t krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:15.484192 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42049
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:15.566015 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:15.566231 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:15.566267 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:15.567317 5947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:15.567389 5947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:15.567412 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:15.567433 5947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:15.567459 5947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:15.568861 5947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:15.569139 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:15.569365 5947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:15.574261 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:15.574318 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:15.574261 5960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:15.574433 5947 server_base.cc:1034] running on GCE node
I20260812 00:16:15.574745 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:15.575029 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:15.576172 5947 hybrid_clock.cc:648] HybridClock initialized: now 1786493775576159 us; error 29 us; skew 500 ppm
Aug 12 00:16:15 dist-test-slave-zr1t krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493775, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:15.578609 5947 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:15.579038 5947 webserver.cc:460] Webserver started at http://127.31.227.62:35303/ using document root <none> and password file <none>
I20260812 00:16:15.579207 5947 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:15.579245 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:15.579327 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:15.580288 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "6c744fb4d7f3466c91a9d5b64d88db8d"
format_stamp: "Formatted at 2026-08-12 00:16:15 on dist-test-slave-zr1t"
server_key: "494f3ff65167c18fcfd1c27f7d5af022"
server_key_iv: "31b9d31839de8f90de1533fef9545f94"
server_key_version: "encryptionkey@0"
I20260812 00:16:15.580585 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "6c744fb4d7f3466c91a9d5b64d88db8d"
format_stamp: "Formatted at 2026-08-12 00:16:15 on dist-test-slave-zr1t"
server_key: "494f3ff65167c18fcfd1c27f7d5af022"
server_key_iv: "31b9d31839de8f90de1533fef9545f94"
server_key_version: "encryptionkey@0"
I20260812 00:16:15.581845 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:16:15.582552 5982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:15.582707 5947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:16:15.582791 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "6c744fb4d7f3466c91a9d5b64d88db8d"
format_stamp: "Formatted at 2026-08-12 00:16:15 on dist-test-slave-zr1t"
server_key: "494f3ff65167c18fcfd1c27f7d5af022"
server_key_iv: "31b9d31839de8f90de1533fef9545f94"
server_key_version: "encryptionkey@0"
I20260812 00:16:15.582867 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:15.585964 5947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:15.588081 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:15.588183 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:15.592244 5947 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:42049
I20260812 00:16:15.592306 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:42049 every 8 connection(s)
I20260812 00:16:15.592636 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:15.593503 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:15.595757 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d: Bootstrap starting.
I20260812 00:16:15.596346 6075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:15.596675 6075 log.cc:824] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:15.597375 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d: No bootstrap required, opened a new log
I20260812 00:16:15.598964 6075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42049 } }
I20260812 00:16:15.599120 6075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:16:15.599166 6075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c744fb4d7f3466c91a9d5b64d88db8d, State: Initialized, Role: FOLLOWER
I20260812 00:16:15.599334 6075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42049 } }
I20260812 00:16:15.599375 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 5947
I20260812 00:16:15.599435 6075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:16:15.599481 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:15.599480 6075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:16:15.599531 6075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:15.599702 32652 external_mini_cluster.cc:1428] Setting key 636515dc7b4deba5e5fbe8555770da08
I20260812 00:16:15.600603 6075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42049 } }
I20260812 00:16:15.600745 6075 leader_election.cc:302] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c744fb4d7f3466c91a9d5b64d88db8d; no voters:
I20260812 00:16:15.600996 6075 leader_election.cc:288] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:16:15.601172 6095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:15.601284 6075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:15.601351 6095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [term 1 LEADER]: Becoming Leader. State: Replica: 6c744fb4d7f3466c91a9d5b64d88db8d, State: Running, Role: LEADER
I20260812 00:16:15.601466 6095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42049 } }
I20260812 00:16:15.601889 6095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c744fb4d7f3466c91a9d5b64d88db8d. Latest consensus state: current_term: 1 leader_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42049 } } }
I20260812 00:16:15.601990 6095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:15.602090 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:15.600207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43647 (local address 127.31.227.62:42049)
0812 00:16:15.600394 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:15.600402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:15.600425 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:16:15.600514 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:15.600519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:15.600541 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:15.600648 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:15.601427 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:15.601431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:15.601431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:15.601436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:15.601488 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:15.601506 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:15.601587 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:15.601758 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 00:16:15.601805 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260812 00:16:15.601907 6098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c744fb4d7f3466c91a9d5b64d88db8d" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 42049 } } }
I20260812 00:16:15.602334 6098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:15.602358 6118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:15.602959 6118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:15.603117 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:15.604867 6118 catalog_manager.cc:1347] Generated new cluster ID: bc4ed61e9b4049b988236b55ea1b9c3a
I20260812 00:16:15.604918 6118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:15.605974 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:15.604724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33377 (local address 127.31.227.62:42049)
0812 00:16:15.604763 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:15.604765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:15.604993 (+ 228us) server_negotiation.cc:407] Connection header received
0812 00:16:15.605055 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:15.605056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:15.605063 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:15.605119 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:15.605732 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:15.605734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:15.605734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:16:15.605739 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:15.605753 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:16:15.605758 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:15.605814 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:15.605869 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 00:16:15.605891 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:16:15.623998 6118 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:15.624204 6118 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:15.631068 6118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c744fb4d7f3466c91a9d5b64d88db8d: Generated new TSK 0
I20260812 00:16:15.631289 6118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:15.641961 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 5947
2026-08-12T00:16:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1558 ms)
[----------] 18 tests from AuthTokenIssuingTest (27972 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:15 dist-test-slave-zr1t krb5kdc[6146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:15 dist-test-slave-zr1t krb5kdc[6146](info): set up 2 sockets
Aug 12 00:16:15 dist-test-slave-zr1t krb5kdc[6146](info): commencing operation
krb5kdc: starting...
W20260812 00:16:16.901901 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:16 dist-test-slave-zr1t krb5kdc[6146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:17.023640 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:33419
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:33419,127.31.227.61:42499,127.31.227.60:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:17.106668 6168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:16:17.106909 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:17.106938 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:17.106974 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:17.108089 6168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:17.108158 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:17.108181 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:17.108202 6168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:17.108229 6168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:17.109759 6168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:33419,127.31.227.61:42499,127.31.227.60:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:17.110039 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:17.110271 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:16:17.115286 6168 server_base.cc:1034] running on GCE node
W20260812 00:16:17.115263 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:17.115255 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:17.115262 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:17.115954 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:17.116258 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:17.117408 6168 hybrid_clock.cc:648] HybridClock initialized: now 1786493777117393 us; error 30 us; skew 500 ppm
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:17.120051 6168 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:17.120509 6168 webserver.cc:460] Webserver started at http://127.31.227.62:45513/ using document root <none> and password file <none>
I20260812 00:16:17.120774 6168 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:17.120821 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:17.120995 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:17.122141 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "ffa04f8bc1bf4a2780990f42b39cd82f"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "735daee77e45d36772458faadd0b5834"
server_key_iv: "83302bfd8fee28cb75e767cf4f1df037"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.122524 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "ffa04f8bc1bf4a2780990f42b39cd82f"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "735daee77e45d36772458faadd0b5834"
server_key_iv: "83302bfd8fee28cb75e767cf4f1df037"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.123859 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:16:17.124660 6200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:17.124882 6168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:16:17.124985 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "ffa04f8bc1bf4a2780990f42b39cd82f"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "735daee77e45d36772458faadd0b5834"
server_key_iv: "83302bfd8fee28cb75e767cf4f1df037"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.125061 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:17.146679 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:17.148751 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:17.148911 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:17.153388 6168 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:33419
I20260812 00:16:17.153460 6307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:33419 every 8 connection(s)
I20260812 00:16:17.153877 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:17.154919 6308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:17.156540 6308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:17.158331 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6168
I20260812 00:16:17.158479 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:17.158710 32652 external_mini_cluster.cc:1428] Setting key 597784cd546ff94d586fa580f721721e
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:17.167447 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.157230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33419 (local address 127.0.0.1:41870)
0812 00:16:17.157412 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.157452 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.157976 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.158301 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.158303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.158646 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.158856 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.158861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.160354 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.160355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.161022 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.161023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.161743 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.161744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.161799 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.162308 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.162320 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.163994 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.165909 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.165915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.165918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.166116 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.166382 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.166383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.166383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.166474 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.166838 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.166844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.166916 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.167023 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.167191 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260812 00:16:17.167450 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.157284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.31.227.62:33419)
0812 00:16:17.157448 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.157463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.157583 (+ 120us) server_negotiation.cc:407] Connection header received
0812 00:16:17.158115 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.158133 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.158171 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.158288 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.158968 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.160256 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.161106 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.161636 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.161753 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.164104 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.164113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.164114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.164128 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.165818 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.166216 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.166222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.166222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.166301 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.166551 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.166552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.166553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.166763 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.166858 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.167071 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.167257 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.31.227.61@KRBTEST.COM; defaulting to no policy
I20260812 00:16:17.168231 6308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:17.168630 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.168519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40454)
0812 00:16:17.168557 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.168606 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260812 00:16:17.168777 6202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
I20260812 00:16:17.169193 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.169087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40456)
0812 00:16:17.169137 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.169162 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 00:16:17.169296 6308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:42499: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab.
Entry for principal kudu/127.31.227.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab.
I20260812 00:16:17.184616 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:42499
--webserver_interface=127.31.227.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab
--principal=kudu/127.31.227.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:33419,127.31.227.61:42499,127.31.227.60:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:16:17.218516 6308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } attempt: 1
I20260812 00:16:17.219079 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.218927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40472)
0812 00:16:17.218999 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.219043 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 00:16:17.219556 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.219463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40478)
0812 00:16:17.219506 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.219535 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 00:16:17.219669 6308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:42499: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
W20260812 00:16:17.267975 6326 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:16:17.268205 6326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:17.268236 6326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:17.268280 6326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:17.269369 6326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:17.269445 6326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:17.269470 6326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:17.269497 6326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:17.269541 6326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:17.270973 6326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:33419,127.31.227.61:42499,127.31.227.60:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab
--principal=kudu/127.31.227.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.227.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:17.271237 6326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:17.271451 6326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:17.276185 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:17.276186 6336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:17.276202 6337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:17.276467 6326 server_base.cc:1034] running on GCE node
I20260812 00:16:17.276713 6326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:17.277036 6326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:17.278205 6326 hybrid_clock.cc:648] HybridClock initialized: now 1786493777278198 us; error 36 us; skew 500 ppm
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:17.280720 6326 init.cc:377] Logged in from keytab as kudu/127.31.227.61@KRBTEST.COM (short username kudu)
I20260812 00:16:17.281131 6326 webserver.cc:460] Webserver started at http://127.31.227.61:33707/ using document root <none> and password file <none>
I20260812 00:16:17.281358 6326 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:17.281432 6326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:17.281596 6326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:17.282601 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data/instance:
uuid: "24f20a817093483a9c44e8ee5f892fec"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "9d1122645988c4901f196af422b39fef"
server_key_iv: "33026e21ca301151d9bd1a37012d5af0"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.282987 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal/instance:
uuid: "24f20a817093483a9c44e8ee5f892fec"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "9d1122645988c4901f196af422b39fef"
server_key_iv: "33026e21ca301151d9bd1a37012d5af0"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.284355 6326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:16:17.285063 6353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:17.285245 6326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:16:17.285347 6326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal
uuid: "24f20a817093483a9c44e8ee5f892fec"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "9d1122645988c4901f196af422b39fef"
server_key_iv: "33026e21ca301151d9bd1a37012d5af0"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.285435 6326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:17.287804 6308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } attempt: 2
I20260812 00:16:17.288204 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.288061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40492)
0812 00:16:17.288123 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.288154 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 00:16:17.288509 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.288414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40494)
0812 00:16:17.288462 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.288485 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 00:16:17.288600 6308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:42499: Network error: Client connection negotiation failed: client connection to 127.31.227.61:42499: connect: Connection refused (error 111)
I20260812 00:16:17.295552 6326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:17.297256 6326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:17.297420 6326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:17.301471 6326 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.61:42499
I20260812 00:16:17.301577 6458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.61:42499 every 8 connection(s)
I20260812 00:16:17.301885 6326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
I20260812 00:16:17.302821 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:17.304283 6460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:17.310015 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6326
I20260812 00:16:17.310143 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-1/wal/instance
I20260812 00:16:17.310377 32652 external_mini_cluster.cc:1428] Setting key b73b084e73a2eeba353340de0899b5c5
I20260812 00:16:17.313194 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.304964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.31.227.62:33419)
0812 00:16:17.305008 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.305009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.305211 (+ 202us) server_negotiation.cc:407] Connection header received
0812 00:16:17.305702 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.305703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.305712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.305765 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.306283 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.306738 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.307680 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.307866 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.307920 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.310155 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.310156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.310156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.310166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.311743 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.312260 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.312262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.312262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.312329 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.312566 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.312567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.312567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.312709 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.312780 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.313009 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.313097 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:17.313216 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.304903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33419 (local address 127.0.0.1:41874)
0812 00:16:17.305090 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.305128 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.305618 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.305816 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.305819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.306035 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.306213 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.306217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.306819 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.306820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.307603 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.307605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.307952 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.307953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.308054 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.308543 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.308552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.310037 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.311830 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.311837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.311840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.312168 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.312400 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.312401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.312401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.312500 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.312775 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.312779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.312884 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.312964 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.313073 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260812 00:16:17.314047 6460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.227.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.60@KRBTEST.COM" created.
I20260812 00:16:17.323259 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.314309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40502)
0812 00:16:17.315990 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.316001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.316048 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.316343 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.316344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.316532 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.316589 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.316591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.317428 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.317429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.317968 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.317970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.318458 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.318459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.318523 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.319002 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.319006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.320166 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.322017 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.322018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.322018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.322183 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.322454 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.322455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.322456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.322522 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.322778 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.322779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.322820 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.322940 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.323073 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670}
I20260812 00:16:17.323276 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.314342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.31.227.61:42499)
0812 00:16:17.315991 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.315995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.316047 (+ 52us) server_negotiation.cc:407] Connection header received
0812 00:16:17.316104 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.316106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.316119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.316219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.316672 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.317327 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.318061 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.318360 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.318457 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.320283 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.320290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.320290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.320295 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.321938 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.322315 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.322317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.322318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.322378 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.322573 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.322573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.322574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.322696 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.322755 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.322995 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.323122 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":1619,"thread_start_us":1596,"threads_started":1}
I20260812 00:16:17.323901 6460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:17.324220 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.324128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52698)
0812 00:16:17.324160 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.324202 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260812 00:16:17.324390 6364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
I20260812 00:16:17.324757 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.324695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52702)
0812 00:16:17.324719 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.324742 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260812 00:16:17.324880 6460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:42305: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab.
Entry for principal kudu/127.31.227.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab.
I20260812 00:16:17.337725 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:42305
--webserver_interface=127.31.227.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab
--principal=kudu/127.31.227.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:33419,127.31.227.61:42499,127.31.227.60:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:16:17.374195 6460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } attempt: 1
I20260812 00:16:17.374717 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.374573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52716)
0812 00:16:17.374641 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.374684 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:16:17.375103 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.375009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52718)
0812 00:16:17.375055 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.375081 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 00:16:17.375216 6460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:42305: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
I20260812 00:16:17.379756 6308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } attempt: 3
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:16:17.387253 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.380190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.31.227.61:42499)
0812 00:16:17.380258 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.380261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.380275 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:16:17.380315 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.380317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.380327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.380399 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.380736 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.381236 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.382056 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.382277 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.382370 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.384628 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.384630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.384630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.384638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.386147 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.386543 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.386544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.386544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.386596 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.386792 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.386793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.386793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.386900 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.386952 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.387118 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.387190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:16:17.387287 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.380099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40504)
0812 00:16:17.380160 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.380173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.380251 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.380405 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.380406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.380585 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.380651 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.380653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.381323 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.381324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.381954 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.381956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.382382 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.382383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.382467 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.382940 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.382946 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.384526 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.386230 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.386231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.386232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.386466 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.386660 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.386661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.386661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.386733 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.386962 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.386963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.387018 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.387090 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.387189 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 00:16:17.387768 6308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:17.388118 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.388009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52730)
0812 00:16:17.388061 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.388093 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:16:17.388473 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.388389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52744)
0812 00:16:17.388432 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.388456 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 00:16:17.388581 6308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:42305: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
I20260812 00:16:17.419806 6308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } attempt: 1
I20260812 00:16:17.420356 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.420216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52746)
0812 00:16:17.420287 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.420329 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 00:16:17.420830 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.420747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52748)
0812 00:16:17.420787 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.420813 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 00:16:17.420946 6308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:42305: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
W20260812 00:16:17.424400 6469 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:16:17.424628 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:17.424669 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:17.424710 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:17.425900 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:17.425961 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:17.425997 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:17.426021 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:17.426048 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:17.427548 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:33419,127.31.227.61:42499,127.31.227.60:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab
--principal=kudu/127.31.227.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.227.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:17.427827 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:17.428102 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:16:17.433080 6469 server_base.cc:1034] running on GCE node
W20260812 00:16:17.433014 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:17.433034 6479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:17.433029 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:17.433466 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:17.433804 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:17.434993 6469 hybrid_clock.cc:648] HybridClock initialized: now 1786493777434984 us; error 42 us; skew 500 ppm
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:17.437583 6469 init.cc:377] Logged in from keytab as kudu/127.31.227.60@KRBTEST.COM (short username kudu)
I20260812 00:16:17.438056 6469 webserver.cc:460] Webserver started at http://127.31.227.60:40139/ using document root <none> and password file <none>
I20260812 00:16:17.438297 6469 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:17.438364 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:17.438522 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:17.439575 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data/instance:
uuid: "f0176887ce3c45adb0c761a3b3684c7a"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "dcc9b2a673c68c76069e37c0b1f8dd2a"
server_key_iv: "a1dc28bc88b378e3bb13fb743c7144e1"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.439949 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal/instance:
uuid: "f0176887ce3c45adb0c761a3b3684c7a"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "dcc9b2a673c68c76069e37c0b1f8dd2a"
server_key_iv: "a1dc28bc88b378e3bb13fb743c7144e1"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.441267 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:16:17.442103 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:17.442261 6469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:17.442351 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal
uuid: "f0176887ce3c45adb0c761a3b3684c7a"
format_stamp: "Formatted at 2026-08-12 00:16:17 on dist-test-slave-zr1t"
server_key: "dcc9b2a673c68c76069e37c0b1f8dd2a"
server_key_iv: "a1dc28bc88b378e3bb13fb743c7144e1"
server_key_version: "encryptionkey@0"
I20260812 00:16:17.442407 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:17.443351 6460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } attempt: 2
I20260812 00:16:17.443729 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.443603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52754)
0812 00:16:17.443663 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.443699 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 00:16:17.444128 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.444040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52758)
0812 00:16:17.444085 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.444111 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 00:16:17.444257 6460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:42305: Network error: Client connection negotiation failed: client connection to 127.31.227.60:42305: connect: Connection refused (error 111)
I20260812 00:16:17.465781 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:17.467653 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:17.467767 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:17.472018 6469 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.60:42305
I20260812 00:16:17.472076 6591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.60:42305 every 8 connection(s)
I20260812 00:16:17.472411 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
I20260812 00:16:17.472608 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6469
I20260812 00:16:17.472728 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493496733958-32652-0/minicluster-data/master-2/wal/instance
I20260812 00:16:17.472955 32652 external_mini_cluster.cc:1428] Setting key f6e3988c59eca65c2cb41dea9bd2f700
I20260812 00:16:17.473755 6592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:17.475849 6592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:17.484220 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.473518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.31.227.60:42305)
0812 00:16:17.473766 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.473777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.473814 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:16:17.474405 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.474415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.474489 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.474634 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.475274 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.476583 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.477572 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.478044 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.478150 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.480568 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.480576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.480577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.480584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.482548 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.482955 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.482957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.482958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.483040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.483259 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.483259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.483260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.483585 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.483726 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.483912 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.484044 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":114,"threads_started":1}
I20260812 00:16:17.484581 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.476607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.31.227.62:33419)
0812 00:16:17.476669 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.476671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.476780 (+ 109us) server_negotiation.cc:407] Connection header received
0812 00:16:17.476844 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.476845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.476855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.476911 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.477291 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.477840 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.478827 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.479154 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.479235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.481697 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.481699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.481699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.481707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.483259 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.483670 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.483671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.483671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.483727 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.483997 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.483998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.483999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.484140 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.484204 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.484444 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.484521 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:16:17.484902 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.476534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:33419 (local address 127.0.0.1:41876)
0812 00:16:17.476687 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.476724 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.476793 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.476924 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.476925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.477157 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.477216 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.477217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.477924 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.477925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.478745 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.478749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.479257 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.479258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.479309 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.480262 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.480269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.481611 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.483333 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.483337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.483339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.483600 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.483843 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.483844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.483844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.483911 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.484222 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.484228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.484278 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.484679 (+ 401us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.484803 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260812 00:16:17.485379 6592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:17.492331 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.485712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.31.227.61:42499)
0812 00:16:17.485787 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.485789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.485799 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:16:17.485894 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.485896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.485904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.485942 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.486195 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.486843 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.487923 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.488181 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.488277 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.490089 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.490090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.490090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.490096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.491355 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.491664 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.491665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.491665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.491717 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.491917 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.491918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.491918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.492021 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.492092 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.492219 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.492274 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:16:17.492369 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.485648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:42499 (local address 127.0.0.1:40520)
0812 00:16:17.485700 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.485711 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.485825 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.485957 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.485958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.486102 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.486127 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.486128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.486930 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.486931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.487800 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.487803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.488292 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.488314 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.488376 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.488626 (+ 250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.488631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.490004 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.491441 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.491442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.491442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.491594 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.491789 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.491790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.491790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.491845 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.492085 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.492086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.492124 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.492185 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.492268 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:16:17.492872 6592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:17.493188 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.485140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.31.227.62:33419)
0812 00:16:17.485175 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.485177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.485279 (+ 102us) server_negotiation.cc:407] Connection header received
0812 00:16:17.485323 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.485323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.485331 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.485397 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.486035 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.486648 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.487608 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.487781 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.487828 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.490828 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.490829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.490830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.490836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.492103 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.492455 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.492456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.492456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.492513 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.492675 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.492676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.492676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.492793 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.492967 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.493030 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.493101 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:17.496090 6308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } attempt: 2
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:16:17.499552 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.493091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52764)
0812 00:16:17.493275 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.493285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.493336 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.493455 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.493455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.493605 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.493634 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.493635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.494282 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.494283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.494738 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.494739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.495114 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.495114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.495158 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.495547 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.495551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.496756 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.498554 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.498556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.498556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.498756 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.498954 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.498955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.498955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.499020 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.499300 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.499301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.499334 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.499399 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.499488 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260812 00:16:17.499830 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.493142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.31.227.60:42305)
0812 00:16:17.493238 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.493239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.493343 (+ 104us) server_negotiation.cc:407] Connection header received
0812 00:16:17.493385 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.493386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.493396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.493444 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.493714 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.494186 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.494829 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.495048 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.495114 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.496833 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.496834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.496834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.496841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.498471 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.498828 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.498829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.498830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.498882 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.499090 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.499091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.499091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.499203 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.499279 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.499677 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.499757 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:16:17.501191 6592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: Bootstrap starting.
I20260812 00:16:17.502020 6592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:17.502221 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.493740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.31.227.61:42499)
0812 00:16:17.493809 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.493811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.493851 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:16:17.493922 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.493922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.493931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.494012 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.494451 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.494881 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.495690 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.495906 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.495986 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.498734 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.498735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.498735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.498742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.500328 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.500698 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.500699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.500700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.500757 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.501482 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.501483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.501483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.501663 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.501815 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.502029 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.502109 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:16:17.502388 6592 log.cc:824] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:17.503525 6592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: No bootstrap required, opened a new log
I20260812 00:16:17.503806 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:17.504714 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.496310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52780)
0812 00:16:17.496353 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.496361 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.496394 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.496723 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.496724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.496874 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.496920 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.496922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.498196 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.498198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.498890 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.498898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.499821 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.499822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.499883 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.500277 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.500282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.501588 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.503584 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.503585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.503586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.503784 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.504047 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.504048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.504048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.504113 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.504390 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.504392 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.504435 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.504521 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.504613 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 00:16:17.504742 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.496431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.31.227.60:42305)
0812 00:16:17.496578 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.496580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.496588 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:16:17.496608 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.496608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.496619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.496697 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.496998 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.498097 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.499263 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.499545 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.500834 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.501652 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.501653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.501654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.501660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.503477 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.503905 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.503906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.503907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.503967 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.504180 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.504181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.504182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.504310 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.504386 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.504547 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.504617 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260812 00:16:17.505600 6592 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.505752 6592 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0176887ce3c45adb0c761a3b3684c7a, State: Initialized, Role: FOLLOWER
I20260812 00:16:17.505841 6592 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.504956 6268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41882
I20260812 00:16:17.506196 6592 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:17.506165 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.506258 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.506387 6308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f: Bootstrap starting.
I20260812 00:16:17.507078 6308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:17.507385 6308 log.cc:824] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:17.508145 6308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f: No bootstrap required, opened a new log
W20260812 00:16:17.508235 6622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:17.508291 6622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:17.509436 6308 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.509595 6308 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa04f8bc1bf4a2780990f42b39cd82f, State: Initialized, Role: FOLLOWER
I20260812 00:16:17.509673 6308 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.509974 6308 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:17.510015 6268 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 00:16:17.510141 6268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:17.508260 6469 master_runner.cc:186] Error getting master registration for 127.31.227.62:33419: OK, instance_id { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" instance_seqno: 1786493777152178 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:16:17.510586 6469 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:16:17.510896 6268 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.511124 6268 leader_election.cc:288] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [CANDIDATE]: Term 1 election: Requested vote from peers 24f20a817093483a9c44e8ee5f892fec (127.31.227.61:42499), f0176887ce3c45adb0c761a3b3684c7a (127.31.227.60:42305)
I20260812 00:16:17.511375 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f0176887ce3c45adb0c761a3b3684c7a"
I20260812 00:16:17.511504 6546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:17.512213 6546 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa04f8bc1bf4a2780990f42b39cd82f in term 1.
I20260812 00:16:17.512431 6204 leader_election.cc:302] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0176887ce3c45adb0c761a3b3684c7a, ffa04f8bc1bf4a2780990f42b39cd82f; no voters:
I20260812 00:16:17.509969 6627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.513042 6662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:17.513089 6627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.513190 6662 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [term 1 LEADER]: Becoming Leader. State: Replica: ffa04f8bc1bf4a2780990f42b39cd82f, State: Running, Role: LEADER
I20260812 00:16:17.513280 6662 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.515295 6627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa04f8bc1bf4a2780990f42b39cd82f. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.515367 6627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:17.513665 6168 master_runner.cc:186] Error getting master registration for 127.31.227.61:42499: OK, instance_id { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" instance_seqno: 1786493777300387 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:16:17.516634 6168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 00:16:17.518150 6644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:17.518199 6644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:17.518251 6670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:17.518385 6670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:17.523166 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.512787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.31.227.61:42499)
0812 00:16:17.512829 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.512831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.513856 (+ 1025us) server_negotiation.cc:407] Connection header received
0812 00:16:17.513992 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.513993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.514008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.514093 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.514760 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.515442 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.517717 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.518581 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.518650 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.520385 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.520386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.520387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.520393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.521926 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.522348 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.522350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.522350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.522404 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.522608 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.522609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.522609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.522725 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.522791 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.523003 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.523076 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:16:17.524312 6546 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [term 1 FOLLOWER]: Refusing update from remote peer ffa04f8bc1bf4a2780990f42b39cd82f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:16:17.524600 6662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:16:17.525853 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa04f8bc1bf4a2780990f42b39cd82f. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.525938 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.525859 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.512631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.31.227.60:42305)
0812 00:16:17.512666 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.512668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.517851 (+ 5183us) server_negotiation.cc:407] Connection header received
0812 00:16:17.518102 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.518103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.518116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.518160 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.518962 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.519655 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.520996 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.521296 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.521335 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.522757 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.522758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.522759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.522765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.524506 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.524924 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.524925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.524926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.524992 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.525207 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.525208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.525208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.525333 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.525409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.525675 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.525762 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:16:17.528424 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.513545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.31.227.62:33419)
0812 00:16:17.513582 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.513585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.518689 (+ 5104us) server_negotiation.cc:407] Connection header received
0812 00:16:17.518712 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.518713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.518722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.518767 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.519764 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.520375 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.521880 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.522160 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.522208 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.525212 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.525213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.525214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.525220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.527183 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.527551 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.527552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.527552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.527606 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.527812 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.527827 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.527828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.527964 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.528047 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.528214 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.528300 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:16:17.535393 6460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } attempt: 3
I20260812 00:16:17.536700 6662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.536787 6662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:17.537230 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.537313 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.538149 6673 mvcc.cc:205] Tried to move back new op lower bound from 7317478512737038336 to 7317478512701153280. Current Snapshot: MvccSnapshot[applied={T|T < 7317478512737038336}]
I20260812 00:16:17.538379 6672 mvcc.cc:205] Tried to move back new op lower bound from 7317478512737038336 to 7317478512701153280. Current Snapshot: MvccSnapshot[applied={T|T < 7317478512737038336}]
Aug 12 00:16:17 dist-test-slave-zr1t krb5kdc[6146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493777, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:16:17.542189 6670 catalog_manager.cc:1347] Generated new cluster ID: 64de9e1613534a22a1d2b9872aae3873
I20260812 00:16:17.542241 6670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:17.543392 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.535608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:42305 (local address 127.0.0.1:52796)
0812 00:16:17.535648 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:17.535657 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:17.535688 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:17.535953 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:17.535954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:17.536102 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:17.536140 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.536142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.537222 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.537223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.538024 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:17.538027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.538516 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.538517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:17.538577 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.538988 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:17.538994 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:17.540365 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:17.542356 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.542357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.542358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.542569 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.542787 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:17.542787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:17.542787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:17.542851 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:17.543137 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:17.543138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:17.543175 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 00:16:17.543226 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:16:17.543305 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 00:16:17.544341 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:17.535771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.31.227.60:42305)
0812 00:16:17.535818 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.535820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.535829 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:16:17.535858 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.535858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.535867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.535924 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.536228 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.536924 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.538108 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:17.538404 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:17.538465 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:17.540467 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:17.540468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:17.540468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:17.540475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:17.542222 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.542656 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.542657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.542657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.542719 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:17.542921 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:17.542921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:17.542922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:17.543044 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:17.543090 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:17.544173 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0812 00:16:17.544262 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:16:17.545953 6460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: Bootstrap starting.
I20260812 00:16:17.546617 6460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:17.546892 6460 log.cc:824] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:17.547833 6460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: No bootstrap required, opened a new log
I20260812 00:16:17.549952 6460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.550091 6460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f20a817093483a9c44e8ee5f892fec, State: Initialized, Role: FOLLOWER
I20260812 00:16:17.550180 6460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } }
I20260812 00:16:17.550536 6460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:17.550652 6410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:17.551515 6410 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [term 1 FOLLOWER]: Refusing update from remote peer ffa04f8bc1bf4a2780990f42b39cd82f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:16:17.552663 6413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24f20a817093483a9c44e8ee5f892fec"
I20260812 00:16:17.553421 6413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa04f8bc1bf4a2780990f42b39cd82f in term 1.
I20260812 00:16:17.553735 6665 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:16:17.555182 6707 mvcc.cc:205] Tried to move back new op lower bound from 7317478512737038336 to 7317478512701153280. Current Snapshot: MvccSnapshot[applied={T|T < 7317478512737038336}]
I20260812 00:16:17.557659 6670 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:17.558394 6670 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:17.561733 6705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.561828 6705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.561893 6705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa04f8bc1bf4a2780990f42b39cd82f. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.561944 6705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.561985 6705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.562027 6705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:17.562741 6627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffa04f8bc1bf4a2780990f42b39cd82f" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 33419 } } peers { permanent_uuid: "24f20a817093483a9c44e8ee5f892fec" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 42499 } } peers { permanent_uuid: "f0176887ce3c45adb0c761a3b3684c7a" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 42305 } } }
I20260812 00:16:17.562835 6627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:17.564415 6704 catalog_manager.cc:1259] Loaded cluster ID: 64de9e1613534a22a1d2b9872aae3873
I20260812 00:16:17.564466 6704 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: loading cluster ID for follower catalog manager: success
I20260812 00:16:17.565621 6704 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: acquiring CA information for follower catalog manager: success
W20260812 00:16:17.566425 6704 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260812 00:16:17.569612 6670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa04f8bc1bf4a2780990f42b39cd82f: Generated new TSK 0
I20260812 00:16:17.569820 6670 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:16:17.603745 6315 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.602114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.31.227.62:33419)
0812 00:16:17.602154 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.602156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.602991 (+ 835us) server_negotiation.cc:407] Connection header received
0812 00:16:17.603106 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.603107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.603121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.603151 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.603660 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41904: BlockingRecv error: recv got EOF from 127.0.0.1:41904 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 00:16:17.604553 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.601891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.31.227.61:42499)
0812 00:16:17.601941 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.601943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.604011 (+ 2068us) server_negotiation.cc:407] Connection header received
0812 00:16:17.604115 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.604116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.604128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.604158 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.604478 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40546: BlockingRecv error: recv got EOF from 127.0.0.1:40546 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 00:16:17.605320 6598 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.602333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.31.227.60:42305)
0812 00:16:17.602374 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.602376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.604793 (+ 2417us) server_negotiation.cc:407] Connection header received
0812 00:16:17.604923 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.604924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.604934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.604964 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.605260 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52812: BlockingRecv error: recv got EOF from 127.0.0.1:52812 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 00:16:17.608033 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.607192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.31.227.61:42499)
0812 00:16:17.607230 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.607232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.607485 (+ 253us) server_negotiation.cc:407] Connection header received
0812 00:16:17.607645 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.607646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.607664 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.607694 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.607987 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40558: BlockingRecv error: recv got EOF from 127.0.0.1:40558 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 00:16:17.611023 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.610229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.31.227.61:42499)
0812 00:16:17.610262 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.610264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.610553 (+ 289us) server_negotiation.cc:407] Connection header received
0812 00:16:17.610647 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.610648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.610669 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.610697 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.610981 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40566: BlockingRecv error: recv got EOF from 127.0.0.1:40566 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 00:16:17.615366 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.614671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.31.227.61:42499)
0812 00:16:17.614706 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.614707 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.614912 (+ 205us) server_negotiation.cc:407] Connection header received
0812 00:16:17.615001 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.615002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.615011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.615038 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.615325 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40574: BlockingRecv error: recv got EOF from 127.0.0.1:40574 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 00:16:17.621590 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.620532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.31.227.61:42499)
0812 00:16:17.620568 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.620569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.621032 (+ 463us) server_negotiation.cc:407] Connection header received
0812 00:16:17.621131 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.621132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.621142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.621172 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.621545 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40578: BlockingRecv error: recv got EOF from 127.0.0.1:40578 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 00:16:17.631788 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.630876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.31.227.61:42499)
0812 00:16:17.630917 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.630919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.631171 (+ 252us) server_negotiation.cc:407] Connection header received
0812 00:16:17.631281 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.631282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.631294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.631325 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.631718 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40592: BlockingRecv error: recv got EOF from 127.0.0.1:40592 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 00:16:17.650297 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.649244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.31.227.61:42499)
0812 00:16:17.649291 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.649294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.649662 (+ 368us) server_negotiation.cc:407] Connection header received
0812 00:16:17.649776 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.649777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.649791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.649823 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.650237 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40596: BlockingRecv error: recv got EOF from 127.0.0.1:40596 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 00:16:17.684819 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.683854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.31.227.61:42499)
0812 00:16:17.683902 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.683904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.684187 (+ 283us) server_negotiation.cc:407] Connection header received
0812 00:16:17.684293 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.684294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.684307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.684339 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.684755 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40606: BlockingRecv error: recv got EOF from 127.0.0.1:40606 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 00:16:17.754841 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.750327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.31.227.61:42499)
0812 00:16:17.750379 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.750382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.754232 (+ 3850us) server_negotiation.cc:407] Connection header received
0812 00:16:17.754275 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.754276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.754294 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.754376 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.754757 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40614: BlockingRecv error: recv got EOF from 127.0.0.1:40614 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:16:17.885048 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:17.884383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.31.227.61:42499)
0812 00:16:17.884451 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:17.884453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:17.884532 (+ 79us) server_negotiation.cc:407] Connection header received
0812 00:16:17.884586 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:17.884587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:17.884598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:17.884664 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:17.884985 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40628: BlockingRecv error: recv got EOF from 127.0.0.1:40628 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260812 00:16:18.143163 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:18.142518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.31.227.61:42499)
0812 00:16:18.142595 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:18.142597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:18.142606 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:16:18.142633 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:18.142634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:18.142644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:18.142696 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:18.143074 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40640: BlockingRecv error: recv got EOF from 127.0.0.1:40640 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:16:18.508775 6622 catalog_manager.cc:1259] Loaded cluster ID: 64de9e1613534a22a1d2b9872aae3873
I20260812 00:16:18.508895 6622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: loading cluster ID for follower catalog manager: success
I20260812 00:16:18.509838 6622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: acquiring CA information for follower catalog manager: success
I20260812 00:16:18.510226 6622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0176887ce3c45adb0c761a3b3684c7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:16:18.566991 6704 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24f20a817093483a9c44e8ee5f892fec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:16:18.659065 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:18.656461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.31.227.61:42499)
0812 00:16:18.656593 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:18.656595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:18.656635 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:16:18.656712 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:18.656713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:18.656778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:18.656843 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:18.656991 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.657475 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.658303 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.658525 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.658583 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:18.658757 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:18.658877 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:18.658937 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:18.658978 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 00:16:18.658995 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260812 00:16:18.663486 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:18.661129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.31.227.60:42305)
0812 00:16:18.661262 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:18.661264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:18.661273 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:16:18.661319 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:18.661320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:18.661375 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:18.661470 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:18.661607 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.662095 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.662796 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.662992 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.663042 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:18.663168 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:18.663265 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:18.663319 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:18.663358 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:16:18.663376 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260812 00:16:18.668006 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:18.665239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.31.227.61:42499)
0812 00:16:18.665284 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:18.665286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:18.665474 (+ 188us) server_negotiation.cc:407] Connection header received
0812 00:16:18.665578 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:18.665579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:18.665590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:18.665657 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:18.665805 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.666485 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.667340 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.667552 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.667629 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:18.667773 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:18.667810 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:18.667862 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:18.667912 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 00:16:18.667935 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:16:18.668243 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:18.665408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.31.227.60:42305)
0812 00:16:18.665453 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:18.665455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:18.665627 (+ 172us) server_negotiation.cc:407] Connection header received
0812 00:16:18.665690 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:18.665691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:18.665701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:18.665751 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:18.665910 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.666471 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.667428 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.667715 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.667751 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:18.667964 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:18.667999 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:18.668032 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:18.668147 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 00:16:18.668169 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:16:18.668982 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:18.665302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.31.227.62:33419)
0812 00:16:18.665441 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:18.665443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:18.665543 (+ 100us) server_negotiation.cc:407] Connection header received
0812 00:16:18.665776 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:18.665778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:18.665830 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:18.665944 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:18.666124 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.666847 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.667811 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:18.668112 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:18.668178 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:18.668424 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:18.668543 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:18.668607 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:18.668892 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 00:16:18.668918 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260812 00:16:18.670957 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6168
I20260812 00:16:18.686081 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6326
I20260812 00:16:18.696494 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6469
2026-08-12T00:16:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3053 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:16:18 dist-test-slave-zr1t krb5kdc[6851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:16:18 dist-test-slave-zr1t krb5kdc[6851](info): set up 2 sockets
Aug 12 00:16:18 dist-test-slave-zr1t krb5kdc[6851](info): commencing operation
krb5kdc: starting...
W20260812 00:16:19.953624 32652 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/test-user.keytab.
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.227.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
Entry for principal kudu/127.31.227.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab.
I20260812 00:16:20.066051 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34935
--webserver_interface=127.31.227.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34935,127.31.227.61:46301,127.31.227.60:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:16:20.145642 6882 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:16:20.145891 6882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:20.145921 6882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:20.145965 6882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:20.147069 6882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:20.147157 6882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:20.147182 6882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:20.147213 6882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:20.147238 6882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:20.148721 6882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34935,127.31.227.61:46301,127.31.227.60:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.62.keytab
--principal=kudu/127.31.227.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.62:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.227.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:20.149003 6882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:20.149232 6882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:20.154285 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:20.154301 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:20.154314 6896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:20.154838 6882 server_base.cc:1034] running on GCE node
I20260812 00:16:20.155161 6882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:20.155460 6882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:20.156615 6882 hybrid_clock.cc:648] HybridClock initialized: now 1786493780156600 us; error 31 us; skew 500 ppm
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:20.162513 6882 init.cc:377] Logged in from keytab as kudu/127.31.227.62@KRBTEST.COM (short username kudu)
I20260812 00:16:20.162995 6882 webserver.cc:460] Webserver started at http://127.31.227.62:40759/ using document root <none> and password file <none>
I20260812 00:16:20.163187 6882 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:20.163229 6882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:20.163327 6882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:20.164407 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data/instance:
uuid: "2653847e8af04c538c64d2357430e592"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "36adb2a3a39a8a87e3769db0a65e7ddd"
server_key_iv: "152437ab8d5fcaa98b561bf602450804"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.164721 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal/instance:
uuid: "2653847e8af04c538c64d2357430e592"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "36adb2a3a39a8a87e3769db0a65e7ddd"
server_key_iv: "152437ab8d5fcaa98b561bf602450804"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.166134 6882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:16:20.166831 6906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:20.167007 6882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:16:20.167120 6882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal
uuid: "2653847e8af04c538c64d2357430e592"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "36adb2a3a39a8a87e3769db0a65e7ddd"
server_key_iv: "152437ab8d5fcaa98b561bf602450804"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.167198 6882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:20.194562 6882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:20.196528 6882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:20.196704 6882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:20.201095 6882 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.62:34935
I20260812 00:16:20.201138 6988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.62:34935 every 8 connection(s)
I20260812 00:16:20.201581 6882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/data/info.pb
I20260812 00:16:20.202672 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:20.204207 6989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:20.210726 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 6882
I20260812 00:16:20.210841 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-0/wal/instance
I20260812 00:16:20.211108 32652 external_mini_cluster.cc:1428] Setting key 1c87988989b0a0adc95cb79a8c7457f7
I20260812 00:16:20.214265 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.204961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.31.227.62:34935)
0812 00:16:20.205122 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.205126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.205194 (+ 68us) server_negotiation.cc:407] Connection header received
0812 00:16:20.205722 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.205724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.205741 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.205827 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.206373 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.207186 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.208044 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.208377 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.208463 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.210579 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.210587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.210588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.210596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.212580 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.212978 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.212982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.212983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.213098 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.213423 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.213423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.213424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.213686 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.213756 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.213999 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.214102 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260812 00:16:20.214354 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.204888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34935 (local address 127.0.0.1:36320)
0812 00:16:20.205067 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.205110 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.205584 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.205868 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.205870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.206125 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.206287 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.206291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.207275 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.207276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.207943 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.207944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.208478 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.208479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.208531 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.209075 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.209084 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.210475 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.212673 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.212680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.212682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.212887 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.213181 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.213182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.213183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.213336 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.213756 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.213763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.213846 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.213963 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.214180 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":57,"thread_start_us":101,"threads_started":1}
I20260812 00:16:20.215061 6989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.215425 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.215288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50518)
0812 00:16:20.215338 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.215391 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 00:16:20.215581 6907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
I20260812 00:16:20.216058 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.215936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50534)
0812 00:16:20.215992 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.216025 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 00:16:20.216234 6989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:46301: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.31.227.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab.
Entry for principal kudu/127.31.227.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab.
I20260812 00:16:20.236195 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:46301
--webserver_interface=127.31.227.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab
--principal=kudu/127.31.227.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34935,127.31.227.61:46301,127.31.227.60:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:16:20.265484 6989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } attempt: 1
I20260812 00:16:20.266070 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.265925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50544)
0812 00:16:20.265990 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.266035 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:16:20.266450 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.266361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50548)
0812 00:16:20.266401 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.266430 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 00:16:20.266561 6989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:46301: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
W20260812 00:16:20.317108 7010 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:16:20.317286 7010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:20.317303 7010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:20.317355 7010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:20.318485 7010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:20.318524 7010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:20.318538 7010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:20.318588 7010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:20.318616 7010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:20.320115 7010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34935,127.31.227.61:46301,127.31.227.60:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.61.keytab
--principal=kudu/127.31.227.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.61:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.227.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:20.320363 7010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:20.320748 7010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:20.325603 7022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:20.325614 7026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:20.325635 7019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:20.325747 7010 server_base.cc:1034] running on GCE node
I20260812 00:16:20.326112 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:20.326385 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:20.327538 7010 hybrid_clock.cc:648] HybridClock initialized: now 1786493780327531 us; error 46 us; skew 500 ppm
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:20.330001 7010 init.cc:377] Logged in from keytab as kudu/127.31.227.61@KRBTEST.COM (short username kudu)
I20260812 00:16:20.330436 7010 webserver.cc:460] Webserver started at http://127.31.227.61:40903/ using document root <none> and password file <none>
I20260812 00:16:20.330638 7010 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:20.330698 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:20.330832 7010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:20.331784 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data/instance:
uuid: "6399c600d64849bab15d9e7a149c6930"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "06e5ab9980468815f26f0c712b7881ad"
server_key_iv: "4544293cb1b476b172209935522844d5"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.332122 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal/instance:
uuid: "6399c600d64849bab15d9e7a149c6930"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "06e5ab9980468815f26f0c712b7881ad"
server_key_iv: "4544293cb1b476b172209935522844d5"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.333361 7010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:16:20.334163 7036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:20.334331 7010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:20.334437 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal
uuid: "6399c600d64849bab15d9e7a149c6930"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "06e5ab9980468815f26f0c712b7881ad"
server_key_iv: "4544293cb1b476b172209935522844d5"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.334511 7010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:20.334708 6989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } attempt: 2
I20260812 00:16:20.335079 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.334968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50556)
0812 00:16:20.335022 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.335054 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 00:16:20.335439 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.335341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50558)
0812 00:16:20.335388 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.335418 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 00:16:20.335528 6989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.61:46301: Network error: Client connection negotiation failed: client connection to 127.31.227.61:46301: connect: Connection refused (error 111)
I20260812 00:16:20.359638 7010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:20.362082 7010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:20.362198 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:20.366277 7010 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.61:46301
I20260812 00:16:20.366364 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.61:46301 every 8 connection(s)
I20260812 00:16:20.366703 7010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/data/info.pb
I20260812 00:16:20.367751 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:20.369346 7130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.371292 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7010
I20260812 00:16:20.371417 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-1/wal/instance
I20260812 00:16:20.371606 32652 external_mini_cluster.cc:1428] Setting key 2ccf81b3aa6ca23fd845265b0152ab87
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:20.379042 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.370147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.31.227.62:34935)
0812 00:16:20.370227 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.370230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.370439 (+ 209us) server_negotiation.cc:407] Connection header received
0812 00:16:20.370896 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.370897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.370908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.370957 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.371723 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.372247 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.373285 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.373635 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.373698 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.376064 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.376066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.376066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.376074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.377626 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.378047 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.378049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.378049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.378114 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.378374 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.378375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.378375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.378501 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.378572 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.378861 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.378949 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:16:20.379235 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.370113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34935 (local address 127.0.0.1:36330)
0812 00:16:20.370299 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.370336 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.370807 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.371077 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.371088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.371327 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.371622 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.371627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.372348 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.372349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.373161 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.373163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.373743 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.373744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.373893 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.374390 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.374400 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.375963 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.377722 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.377730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.377732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.377968 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.378189 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.378189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.378190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.378298 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.378576 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.378584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.378722 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.378825 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.378989 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260812 00:16:20.379878 7130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.227.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.227.60@KRBTEST.COM" created.
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:16:20.387513 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.380165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:50570)
0812 00:16:20.380409 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.380421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.380471 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.380611 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.380612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.380742 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.380792 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.380793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.381756 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.381756 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.382239 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.382240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.382785 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.382786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.382873 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.383255 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.383261 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.384655 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.386445 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.386446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.386446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.386609 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.386829 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.386829 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.386830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.386884 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.387182 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.387183 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.387223 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.387289 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.387396 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260812 00:16:20.387540 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.380243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.31.227.61:46301)
0812 00:16:20.380442 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.380446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.380466 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:16:20.380530 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.380532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.380543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.380604 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.380855 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.381687 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.382328 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.382691 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.382787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.384742 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.384751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.384751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.384759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.386338 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.386702 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.386707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.386712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.386761 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.386958 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.386959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.386960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.387101 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.387194 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.387338 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.387449 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260812 00:16:20.388141 7130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.388510 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.388385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51598)
0812 00:16:20.388437 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.388482 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:16:20.388687 7040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
I20260812 00:16:20.389094 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.389030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51612)
0812 00:16:20.389055 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.389078 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260812 00:16:20.389220 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:43205: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.227.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab.
Entry for principal kudu/127.31.227.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab.
I20260812 00:16:20.398097 32652 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkgTkG/build/release/bin/kudu
/tmp/dist-test-taskSkgTkG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:43205
--webserver_interface=127.31.227.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab
--principal=kudu/127.31.227.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.227.20:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.62:34935,127.31.227.61:46301,127.31.227.60:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:16:20.426676 6989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } attempt: 3
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:16:20.433750 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.427060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:43202)
0812 00:16:20.427130 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.427149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.427223 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.427467 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.427469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.427638 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.427705 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.427707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.428378 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.428380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.428923 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.428925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.429330 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.429331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.429412 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.429889 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.429894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.431084 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.432799 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.432800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.432800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.432961 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.433168 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.433168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.433168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.433219 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.433472 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.433472 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.433510 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.433606 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.433684 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 00:16:20.433827 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.427135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.31.227.61:46301)
0812 00:16:20.427205 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.427208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.427220 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:16:20.427379 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.427380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.427390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.427459 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.427791 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.428292 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.429013 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.429253 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.429332 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.431169 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.431171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.431171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.431179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.432706 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.433031 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.433033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.433033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.433097 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.433278 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.433279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.433279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.433412 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.433477 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.433639 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.433720 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:16:20.434226 6989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.434583 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.434468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51622)
0812 00:16:20.434523 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.434560 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:16:20.434897 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.434806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51630)
0812 00:16:20.434849 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.434874 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 00:16:20.434978 6989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:43205: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
I20260812 00:16:20.438400 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } attempt: 1
I20260812 00:16:20.438766 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.438656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51644)
0812 00:16:20.438708 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.438742 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 00:16:20.439119 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.439036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51646)
0812 00:16:20.439079 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.439104 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 00:16:20.439224 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:43205: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
I20260812 00:16:20.466113 6989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } attempt: 1
I20260812 00:16:20.466723 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.466556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51658)
0812 00:16:20.466626 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.466680 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:16:20.467147 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.467035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51668)
0812 00:16:20.467088 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.467122 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 00:16:20.467290 6989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:43205: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
W20260812 00:16:20.484486 7138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:16:20.484706 7138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:16:20.484746 7138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:16:20.484786 7138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:16:20.485963 7138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:16:20.486018 7138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:16:20.486043 7138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:16:20.486073 7138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:16:20.486100 7138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:16:20.487524 7138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.20:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.227.62:34935,127.31.227.61:46301,127.31.227.60:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/krb5kdc/kudu_127.31.227.60.keytab
--principal=kudu/127.31.227.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.60:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.227.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:58:32 UTC on ccycloud.quasar-jwjgej.root.comops.site
build id 4581605
I20260812 00:16:20.487794 7138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:16:20.488013 7138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:16:20.492702 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:16:20.492846 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:20.492973 7138 server_base.cc:1034] running on GCE node
W20260812 00:16:20.492739 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:16:20.493281 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:16:20.493624 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:16:20.494786 7138 hybrid_clock.cc:648] HybridClock initialized: now 1786493780494779 us; error 34 us; skew 500 ppm
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:16:20.497267 7138 init.cc:377] Logged in from keytab as kudu/127.31.227.60@KRBTEST.COM (short username kudu)
I20260812 00:16:20.497701 7138 webserver.cc:460] Webserver started at http://127.31.227.60:40477/ using document root <none> and password file <none>
I20260812 00:16:20.497895 7138 fs_manager.cc:359] Metadata directory not provided
I20260812 00:16:20.497941 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:16:20.498027 7138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:16:20.498941 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data/instance:
uuid: "6d40e585fcc9428581c6d5e4597effea"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "1454ce336cbb46fe8f667c6c61795db5"
server_key_iv: "7118f0a82eb33285a994ed0c8116bee4"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.499231 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal/instance:
uuid: "6d40e585fcc9428581c6d5e4597effea"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "1454ce336cbb46fe8f667c6c61795db5"
server_key_iv: "7118f0a82eb33285a994ed0c8116bee4"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.500435 7138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:16:20.501078 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:16:20.501230 7138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:16:20.501312 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data,/tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal
uuid: "6d40e585fcc9428581c6d5e4597effea"
format_stamp: "Formatted at 2026-08-12 00:16:20 on dist-test-slave-zr1t"
server_key: "1454ce336cbb46fe8f667c6c61795db5"
server_key_iv: "7118f0a82eb33285a994ed0c8116bee4"
server_key_version: "encryptionkey@0"
I20260812 00:16:20.501362 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:16:20.507369 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } attempt: 2
I20260812 00:16:20.507732 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.507614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51670)
0812 00:16:20.507670 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.507705 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:16:20.508124 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.508043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51680)
0812 00:16:20.508082 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.508106 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 00:16:20.508221 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.227.60:43205: Network error: Client connection negotiation failed: client connection to 127.31.227.60:43205: connect: Connection refused (error 111)
I20260812 00:16:20.513052 7138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:16:20.514859 7138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:16:20.515010 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:16:20.518854 7138 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.60:43205
I20260812 00:16:20.518913 7267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.60:43205 every 8 connection(s)
I20260812 00:16:20.519230 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/data/info.pb
I20260812 00:16:20.520200 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:16:20.521651 7270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.523379 32652 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkgTkG/build/release/bin/kudu as pid 7138
I20260812 00:16:20.523509 32652 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkgTkG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493496733958-32652-0/minicluster-data/master-2/wal/instance
I20260812 00:16:20.523706 32652 external_mini_cluster.cc:1428] Setting key 3e7ee41946916cd4a54c56464b53779f
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:16:20.531383 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.522308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.31.227.62:34935)
0812 00:16:20.522372 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.522374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.522575 (+ 201us) server_negotiation.cc:407] Connection header received
0812 00:16:20.523002 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.523003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.523011 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.523062 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.523883 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.524625 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.525877 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.526173 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.526221 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.528408 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.528410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.528410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.528424 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.530016 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.530402 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.530403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.530404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.530462 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.530725 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.530726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.530727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.530835 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.530924 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.531188 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.531298 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:16:20.531559 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.522274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.62:34935 (local address 127.0.0.1:55008)
0812 00:16:20.522453 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.522490 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.522925 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.523106 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.523109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.523452 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.523728 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.523733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.524716 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.524718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.525800 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.525802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.526269 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.526270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.526398 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.526852 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.526861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.528318 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.530098 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.530105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.530107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.530307 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.530530 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.530531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.530532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.530632 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.530920 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.530927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.531050 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.531150 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.531332 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260812 00:16:20.532230 7270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.532687 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.524100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.31.227.60:43205)
0812 00:16:20.524289 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.524293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.524333 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:16:20.524388 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.524389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.524409 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.524485 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.525207 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.526041 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.526996 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.527270 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.527322 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.529871 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.529879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.529879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.529887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.531507 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.531853 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.531856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.531856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.531929 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.532139 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.532140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.532141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.532281 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.532450 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.532504 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.532578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.62@KRBTEST.COM
I20260812 00:16:20.539318 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.532542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.31.227.61:46301)
0812 00:16:20.532584 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.532586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.532593 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:16:20.532620 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.532620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.532630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.532688 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.532942 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.533569 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.534422 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.534722 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.534782 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.536550 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.536551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.536552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.536558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.538126 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.538562 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.538563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.538564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.538626 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.538832 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.538833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.538833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.538930 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.539011 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.539184 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.539262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:16:20.539333 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.532491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.61:46301 (local address 127.0.0.1:43210)
0812 00:16:20.532528 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.532536 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.532569 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.532697 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.532698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.532822 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.532848 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.532849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.533632 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.533633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.534336 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.534338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.534791 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.534792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.534847 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.535169 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.535174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.536466 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.538238 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.538239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.538240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.538491 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.538696 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.538697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.538698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.538764 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.539023 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.539024 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.539058 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.539140 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.539224 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260812 00:16:20.539816 7270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:16:20.540957 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.533416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.31.227.62:34935)
0812 00:16:20.533452 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.533453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.533475 (+ 22us) server_negotiation.cc:407] Connection header received
0812 00:16:20.533493 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.533494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.533501 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.533587 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.534143 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.534676 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.535661 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.535828 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.535874 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.538249 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.538250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.538250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.538257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.539837 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.540280 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.540281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.540282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.540340 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.540510 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.540510 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.540511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.540640 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.540744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.540843 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.540897 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:16:20.542448 6989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } attempt: 2
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.60@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.227.61@KRBTEST.COM
I20260812 00:16:20.547888 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.540039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51698)
0812 00:16:20.540086 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.540095 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.540128 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.540269 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.540270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.540428 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.540458 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.540459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.541489 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.541491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.542110 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.542112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.542901 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.542904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.542949 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.543302 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.543305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.544644 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.546856 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.546858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.546858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.547083 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.547302 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.547303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.547303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.547363 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.547602 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.547602 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.547640 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.547702 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.547793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.62@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:16:20.548168 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.540080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.31.227.60:43205)
0812 00:16:20.540112 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.540114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.540122 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:16:20.540174 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.540175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.540185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.540252 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.540534 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.541430 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.542217 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.542524 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.542893 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.544735 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.544736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.544736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.544742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.546781 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.547182 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.547183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.547183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.547240 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.547429 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.547430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.547430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.547544 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.547608 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.547730 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.547839 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":15}
I20260812 00:16:20.549485 7270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: Bootstrap starting.
I20260812 00:16:20.549870 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.541587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.31.227.61:46301)
0812 00:16:20.541621 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.541623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.541631 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:16:20.541679 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.541680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.541688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.541742 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.542182 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.542772 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.543613 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.543902 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.543978 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.546532 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.546534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.546535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.546541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.548348 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.548714 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.548715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.548715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.548771 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.548969 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.548970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.548970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.549093 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.549251 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.549609 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.549689 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:16:20.550257 7270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:20.550531 7270 log.cc:824] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:20.551319 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.542693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51700)
0812 00:16:20.542736 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.542745 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.542782 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.543766 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.543768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.543922 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.543961 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.543962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.544835 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.544837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.545481 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.545484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.546479 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.546481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.546537 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.546926 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.546930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.548230 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.550298 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.550299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.550300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.550468 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.550662 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.550663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.550663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.550722 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.551005 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.551006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.551048 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.551133 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.551223 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 00:16:20.551353 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.542741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.31.227.60:43205)
0812 00:16:20.543639 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.543641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.543646 (+ 5us) server_negotiation.cc:407] Connection header received
0812 00:16:20.543661 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.543662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.543669 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.543759 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.544069 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.544724 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.545623 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.546398 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.546480 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.548321 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.548322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.548322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.548329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.550198 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.550541 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.550542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.550542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.550602 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.550774 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.550775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.550776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.550943 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.551010 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.551158 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.551226 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":75,"threads_started":1}
I20260812 00:16:20.551652 32652 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:16:20.551935 7270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: No bootstrap required, opened a new log
I20260812 00:16:20.552431 6951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2653847e8af04c538c64d2357430e592"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55014
I20260812 00:16:20.553205 6989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592: Bootstrap starting.
I20260812 00:16:20.553881 6989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:20.554062 7270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.554239 7270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d40e585fcc9428581c6d5e4597effea, State: Initialized, Role: FOLLOWER
I20260812 00:16:20.554241 6989 log.cc:824] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:20.554332 7270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.554641 7270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:20.554594 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.554703 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:20.555006 6989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592: No bootstrap required, opened a new log
I20260812 00:16:20.556306 6989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
W20260812 00:16:20.556411 7301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:20.556456 7301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:20.556452 6989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2653847e8af04c538c64d2357430e592, State: Initialized, Role: FOLLOWER
I20260812 00:16:20.556521 6989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.556824 6989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:20.556849 6951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 00:16:20.557579 6951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:20.556663 7138 master_runner.cc:186] Error getting master registration for 127.31.227.62:34935: OK, instance_id { permanent_uuid: "2653847e8af04c538c64d2357430e592" instance_seqno: 1786493780199839 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:16:20.558322 6951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.558360 7138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:16:20.558570 6951 leader_election.cc:288] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [CANDIDATE]: Term 1 election: Requested vote from peers 6399c600d64849bab15d9e7a149c6930 (127.31.227.61:46301), 6d40e585fcc9428581c6d5e4597effea (127.31.227.60:43205)
I20260812 00:16:20.558812 7216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2653847e8af04c538c64d2357430e592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d40e585fcc9428581c6d5e4597effea"
I20260812 00:16:20.558924 7216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:20.559754 7216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2653847e8af04c538c64d2357430e592 in term 1.
W20260812 00:16:20.560298 7330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:20.560357 7330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:20.559971 6910 leader_election.cc:302] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2653847e8af04c538c64d2357430e592, 6d40e585fcc9428581c6d5e4597effea; no voters:
I20260812 00:16:20.561193 6882 master_runner.cc:186] Error getting master registration for 127.31.227.61:46301: OK, instance_id { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" instance_seqno: 1786493780365073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:16:20.564025 6882 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:16:20.565675 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.565765 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:20.565810 7305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:16:20.565850 7305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [term 1 LEADER]: Becoming Leader. State: Replica: 2653847e8af04c538c64d2357430e592, State: Running, Role: LEADER
I20260812 00:16:20.565910 7305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.566336 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2653847e8af04c538c64d2357430e592. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.566428 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:20.568145 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.559597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.31.227.60:43205)
0812 00:16:20.559628 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.559630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.560560 (+ 930us) server_negotiation.cc:407] Connection header received
0812 00:16:20.560658 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.560659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.560669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.560718 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.561416 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.562102 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.563144 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.563433 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.563498 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.565108 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.565109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.565110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.565117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.566856 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.567309 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.567311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.567311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.567367 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.567572 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.567573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.567573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.567683 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.567759 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.567982 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.568058 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 00:16:20.569900 7338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:16:20.570128 7338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:16:20.577005 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.566707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.31.227.62:34935)
0812 00:16:20.566741 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.566743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.567298 (+ 555us) server_negotiation.cc:407] Connection header received
0812 00:16:20.567460 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.567461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.567472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.567501 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.568589 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.569253 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.571232 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.571524 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.571568 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.573835 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.573837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.573837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.573844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.575432 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.575983 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.575984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.575985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.576040 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.576328 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.576329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.576329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.576463 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.576540 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.576847 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.576922 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:16:20.577005 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.567010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.31.227.61:46301)
0812 00:16:20.567044 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.567046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.568682 (+ 1636us) server_negotiation.cc:407] Connection header received
0812 00:16:20.568778 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.568779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.568788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.568817 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.569447 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.570218 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.571264 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.572268 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.572314 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.573768 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.573770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.573770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.573777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.575835 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.576224 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.576226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.576226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.576278 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.576472 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.576473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.576473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.576585 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.576646 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.576833 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.576917 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 00:16:20.578518 7216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [term 1 FOLLOWER]: Refusing update from remote peer 2653847e8af04c538c64d2357430e592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:16:20.578814 7305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:16:20.579983 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2653847e8af04c538c64d2357430e592. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.580080 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:20.581478 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.581653 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: This master's current role is: LEADER
I20260812 00:16:20.581732 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.581808 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:20.581908 7344 mvcc.cc:205] Tried to move back new op lower bound from 7317478525246267392 to 7317478525198688256. Current Snapshot: MvccSnapshot[applied={T|T < 7317478525246267392}]
I20260812 00:16:20.587625 7338 catalog_manager.cc:1347] Generated new cluster ID: d0f84ba5e2804d54a8384cfd5da2b0bd
I20260812 00:16:20.587674 7338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:16:20.599337 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } attempt: 3
Aug 12 00:16:20 dist-test-slave-zr1t krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.227.61@KRBTEST.COM for kudu/127.31.227.60@KRBTEST.COM
I20260812 00:16:20.608256 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.599596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.60:43205 (local address 127.0.0.1:51728)
0812 00:16:20.599672 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0812 00:16:20.599683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:16:20.599717 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:16:20.599970 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:16:20.599971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:16:20.600141 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:16:20.600186 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.600187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.601141 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.601142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.602340 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:16:20.602344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.602841 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.602842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:16:20.602905 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.603381 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:16:20.603386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:16:20.605136 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:16:20.607064 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.607065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.607066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.607281 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.607524 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:16:20.607524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:16:20.607525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:16:20.607593 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:16:20.607908 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:16:20.607909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:16:20.607987 (+ 78us) client_negotiation.cc:769] Sending connection context
0812 00:16:20.608049 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 00:16:20.608155 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 00:16:20.609321 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:20.599786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.31.227.60:43205)
0812 00:16:20.599821 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.599823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.599830 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:16:20.599850 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.599850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.599859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.599928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.600268 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.600978 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.602440 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:20.602733 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:20.602790 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:20.605249 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:16:20.605250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:16:20.605251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:16:20.605258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:16:20.606944 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.607371 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.607372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.607372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.607442 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:16:20.607670 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:16:20.607670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:16:20.607671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:16:20.607804 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:16:20.607858 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:20.609142 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0812 00:16:20.609230 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:16:20.609755 7338 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:16:20.610605 7338 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:16:20.611234 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: Bootstrap starting.
I20260812 00:16:20.611974 7130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: Neither blocks nor log segments found. Creating new log.
I20260812 00:16:20.612273 7130 log.cc:824] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: Log is configured to *not* fsync() on all Append() calls
I20260812 00:16:20.616127 7338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592: Generated new TSK 0
I20260812 00:16:20.616312 7338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:16:20.618192 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: No bootstrap required, opened a new log
I20260812 00:16:20.621865 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.622017 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6399c600d64849bab15d9e7a149c6930, State: Initialized, Role: FOLLOWER
I20260812 00:16:20.622152 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } }
I20260812 00:16:20.622603 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:16:20.622678 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2653847e8af04c538c64d2357430e592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6399c600d64849bab15d9e7a149c6930"
I20260812 00:16:20.622781 7084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:16:20.623672 7084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2653847e8af04c538c64d2357430e592 in term 1.
I20260812 00:16:20.623888 7081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [term 1 FOLLOWER]: Refusing update from remote peer 2653847e8af04c538c64d2357430e592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:16:20.624217 7336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:16:20.622582 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.629153 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:20.629375 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.629498 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:16:20.630071 7374 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:16:20.630133 7374 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:16:20.631189 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2653847e8af04c538c64d2357430e592. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:20.631290 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:16:20.631863 7378 mvcc.cc:205] Tried to move back new op lower bound from 7317478525398077440 to 7317478525198688256. Current Snapshot: MvccSnapshot[applied={T|T < 7317478525246267392}]
W20260812 00:16:20.650650 7279 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.648845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.31.227.60:43205)
0812 00:16:20.648895 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.648898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.649818 (+ 920us) server_negotiation.cc:407] Connection header received
0812 00:16:20.650050 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.650052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.650064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.650096 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.650580 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51742: BlockingRecv error: recv got EOF from 127.0.0.1:51742 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:16:20.651574 6995 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.649269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.31.227.62:34935)
0812 00:16:20.649313 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.649314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.651043 (+ 1729us) server_negotiation.cc:407] Connection header received
0812 00:16:20.651142 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.651143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.651155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.651185 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.651511 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55026: BlockingRecv error: recv got EOF from 127.0.0.1:55026 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 00:16:20.652415 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.649649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.31.227.61:46301)
0812 00:16:20.649688 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.649690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.651887 (+ 2197us) server_negotiation.cc:407] Connection header received
0812 00:16:20.652012 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.652013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.652023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.652052 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.652355 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43242: BlockingRecv error: recv got EOF from 127.0.0.1:43242 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 00:16:20.654896 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.653767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.31.227.61:46301)
0812 00:16:20.653801 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.653802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.654382 (+ 580us) server_negotiation.cc:407] Connection header received
0812 00:16:20.654504 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.654505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.654524 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.654552 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.654849 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43256: BlockingRecv error: recv got EOF from 127.0.0.1:43256 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 00:16:20.658057 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.657251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.31.227.61:46301)
0812 00:16:20.657283 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.657284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.657570 (+ 286us) server_negotiation.cc:407] Connection header received
0812 00:16:20.657663 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.657664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.657674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.657702 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.658015 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43262: BlockingRecv error: recv got EOF from 127.0.0.1:43262 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 00:16:20.662063 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.661269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.31.227.61:46301)
0812 00:16:20.661305 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.661307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.661562 (+ 255us) server_negotiation.cc:407] Connection header received
0812 00:16:20.661655 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.661655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.661666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.661696 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.662022 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43268: BlockingRecv error: recv got EOF from 127.0.0.1:43268 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 00:16:20.668069 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.667326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.31.227.61:46301)
0812 00:16:20.667372 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.667374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.667577 (+ 203us) server_negotiation.cc:407] Connection header received
0812 00:16:20.667666 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.667667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.667678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.667706 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.668025 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43278: BlockingRecv error: recv got EOF from 127.0.0.1:43278 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 00:16:20.678416 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.677482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.31.227.61:46301)
0812 00:16:20.677522 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.677555 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.677822 (+ 267us) server_negotiation.cc:407] Connection header received
0812 00:16:20.677932 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.677933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.677945 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.677976 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.678366 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43280: BlockingRecv error: recv got EOF from 127.0.0.1:43280 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 00:16:20.697007 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.696081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.31.227.61:46301)
0812 00:16:20.696126 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.696128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.696399 (+ 271us) server_negotiation.cc:407] Connection header received
0812 00:16:20.696510 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.696512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.696524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.696557 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.696950 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43284: BlockingRecv error: recv got EOF from 127.0.0.1:43284 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 00:16:20.731542 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.730571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.31.227.61:46301)
0812 00:16:20.730619 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.730622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.730882 (+ 260us) server_negotiation.cc:407] Connection header received
0812 00:16:20.731022 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.731023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.731035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.731065 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.731480 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43288: BlockingRecv error: recv got EOF from 127.0.0.1:43288 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:16:20.798033 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.797310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.31.227.61:46301)
0812 00:16:20.797373 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.797376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.797534 (+ 158us) server_negotiation.cc:407] Connection header received
0812 00:16:20.797570 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.797571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.797585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.797645 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.797978 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43294: BlockingRecv error: recv got EOF from 127.0.0.1:43294 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 00:16:20.928043 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:20.927176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.31.227.61:46301)
0812 00:16:20.927273 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:20.927276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:20.927403 (+ 127us) server_negotiation.cc:407] Connection header received
0812 00:16:20.927508 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:20.927509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:20.927521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:20.927599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:20.927973 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43308: BlockingRecv error: recv got EOF from 127.0.0.1:43308 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 00:16:21.176115 7482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2653847e8af04c538c64d2357430e592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2653847e8af04c538c64d2357430e592" member_type: VOTER last_known_addr { host: "127.31.227.62" port: 34935 } } peers { permanent_uuid: "6399c600d64849bab15d9e7a149c6930" member_type: VOTER last_known_addr { host: "127.31.227.61" port: 46301 } } peers { permanent_uuid: "6d40e585fcc9428581c6d5e4597effea" member_type: VOTER last_known_addr { host: "127.31.227.60" port: 43205 } } }
I20260812 00:16:21.176252 7482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2653847e8af04c538c64d2357430e592 [sys.catalog]: This master's current role is: LEADER
W20260812 00:16:21.185923 7135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:16:21.185224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.31.227.61:46301)
0812 00:16:21.185306 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:21.185308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:21.185320 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:16:21.185395 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:21.185396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:21.185407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:21.185463 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:16:21.185850 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43324: BlockingRecv error: recv got EOF from 127.0.0.1:43324 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 00:16:21.557173 7301 catalog_manager.cc:1259] Loaded cluster ID: d0f84ba5e2804d54a8384cfd5da2b0bd
I20260812 00:16:21.557327 7301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: loading cluster ID for follower catalog manager: success
I20260812 00:16:21.558820 7301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: acquiring CA information for follower catalog manager: success
I20260812 00:16:21.559255 7301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d40e585fcc9428581c6d5e4597effea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:16:21.630821 7374 catalog_manager.cc:1259] Loaded cluster ID: d0f84ba5e2804d54a8384cfd5da2b0bd
I20260812 00:16:21.630939 7374 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: loading cluster ID for follower catalog manager: success
I20260812 00:16:21.632018 7374 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: acquiring CA information for follower catalog manager: success
I20260812 00:16:21.632462 7374 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6399c600d64849bab15d9e7a149c6930: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:16:21.701789 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:21.699164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.31.227.61:46301)
0812 00:16:21.699299 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:21.699302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:21.699396 (+ 94us) server_negotiation.cc:407] Connection header received
0812 00:16:21.699458 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:21.699459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:21.699513 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:21.699581 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:21.699729 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.700230 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.700988 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.701179 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.701229 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:21.701371 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:21.701481 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:21.701608 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:21.701644 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 00:16:21.701664 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260812 00:16:21.706643 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:21.703680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.31.227.60:43205)
0812 00:16:21.703827 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:21.703829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:21.703838 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:16:21.703919 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:21.703920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:21.703957 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:21.704042 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:21.704181 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.704710 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.705597 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.705862 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.705948 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:21.706134 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:21.706294 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:21.706378 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:21.706433 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 00:16:21.706460 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260812 00:16:21.713155 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:21.709072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.31.227.60:43205)
0812 00:16:21.709284 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:21.709286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:21.709295 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:16:21.709431 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:21.709433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:21.709457 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:21.709665 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:21.709928 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.710575 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.711921 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.712254 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.712414 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:21.712672 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:21.712722 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:21.712790 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:21.712930 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 00:16:21.712966 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260812 00:16:21.713354 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:21.709124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.31.227.61:46301)
0812 00:16:21.709187 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:21.709189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:21.709228 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:16:21.709395 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:21.709396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:21.709408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:21.709559 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:21.709800 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.710848 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.712056 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.712489 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.712596 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:21.712832 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:21.712876 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:21.712959 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:21.713142 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 00:16:21.713193 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:16:21.714409 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:16:21.708836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.31.227.62:34935)
0812 00:16:21.709024 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 00:16:21.709026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:16:21.709223 (+ 197us) server_negotiation.cc:407] Connection header received
0812 00:16:21.709475 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:16:21.709477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:16:21.709591 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:16:21.709726 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:16:21.710077 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.711104 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.712563 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:16:21.713256 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:16:21.713374 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:16:21.713856 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:16:21.714016 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:16:21.714071 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:16:21.714219 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 00:16:21.714251 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260812 00:16:21.717125 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 6882
I20260812 00:16:21.739032 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7010
I20260812 00:16:21.753603 32652 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkgTkG/build/release/bin/kudu with pid 7138
2026-08-12T00:16:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3066 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6120 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285043 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST